Publications

Liao, Huafei L.; Groth, Katrina G.; Adams, Susan S.

Abstract not provided.

Adams, Susan S.; Wenner, Caren

Workplace safety has been historically neglected by organizations in order to enhance profitability. Over the past 30 years, safety concerns and attention to safety have increased due to a series of disastrous events occurring across many different industries (e.g., Chernobyl, Upper Big-Branch Mine, Davis-Besse etc.). Many organizations have focused on promoting a healthy safety culture as a way to understand past incidents, and to prevent future disasters. There is an extensive academic literature devoted to safety culture, and the Department of Energy has also published a significant number of documents related to safety culture. The purpose of the current endeavor was to conduct a review of the safety culture literature in order to understand definitions, methodologies, models, and successful interventions for improving safety culture. After reviewing the literature, we observed four emerging themes. First, it was apparent that although safety culture is a valuable construct, it has some inherent weaknesses. For example, there is no common definition of safety culture and no standard way for assessing the construct. Second, it is apparent that researchers know how to measure particular components of safety culture, with specific focus on individual and organizational factors. Such existing methodologies can be leveraged for future assessments. Third, based on the published literature, the relationship between safety culture and performance is tenuous at best. There are few empirical studies that examine the relationship between safety culture and safety performance metrics. Further, most of these studies do not include a description of the implementation of interventions to improve safety culture, or do not measure the effect of these interventions on safety culture or performance. Fourth, safety culture is best viewed as a dynamic, multi-faceted overall system composed of individual, engineered and organizational models. By addressing all three components of safety culture, organizations have a better chance of understanding, evaluating, and making positive changes towards safety within their own organization.

Forsythe, James C.; Adams, Susan S.; Silva, Austin R.

Within cyber security, the human element represents one of the greatest untapped opportunities for increasing the effectiveness of network defenses. However, there has been little research to understand the human dimension in cyber operations. To better understand the needs and priorities for research and development to address these issues, a workshop was conducted August 28-29, 2012 in Washington DC. A synthesis was developed that captured the key issues and associated research questions. Research and development needs were identified that fell into three parallel paths: (1) human factors analysis and scientific studies to establish foundational knowledge concerning factors underlying the performance of cyber defenders; (2) development of models that capture key processes that mediate interactions between defenders, users, adversaries and the public; and (3) development of a multi-purpose test environment for conducting controlled experiments that enables systems and human performance measurement. These research and development investments would transform cyber operations from an art to a science, enabling systems solutions to be engineered to address a range of situations. Organizations would be able to move beyond the current state where key decisions (e.g. personnel assignment) are made on a largely ad hoc basis to a state in which there exist institutionalized processes for assuring the right people are doing the right jobs in the right way. These developments lay the groundwork for emergence of a professional class of cyber defenders with defined roles and career progressions, with higher levels of personnel commitment and retention. Finally, the operational impact would be evident in improved performance, accompanied by a shift to a more proactive response in which defenders have the capacity to exert greater control over the cyber battlespace.

Matzen, Laura E.; Haass, Michael J.; Trumbo, Michael C.; Adams, Susan S.; Silva, Austin R.

Abstract not provided.

Cole, Kerstan S.; Trumbo, Michael C.; Adams, Susan S.; Matzen, Laura E.; Forsythe, James C.

Abstract not provided.

Trumbo, Michael C.; Haass, Michael J.; Adams, Susan S.; Hendrickson, Stacey M.; Abbott, Robert G.

This report summarizes research conducted through the Sandia National Laboratories Robust Automated Knowledge Capture Laboratory Directed Research and Development project. The objective of this project was to advance scientific understanding of the influence of individual cognitive attributes on decision making. The project has developed a quantitative model known as RumRunner that has proven effective in predicting the propensity of an individual to shift strategies on the basis of task and experience related parameters. Three separate studies are described which have validated the basic RumRunner model. This work provides a basis for better understanding human decision making in high consequent national security applications, and in particular, the individual characteristics that underlie adaptive thinking.

Communications in Computer and Information Science

Bandlow, Alisa; Matzen, Laura E.; Cole, Kerstan S.; Dornburg, Courtney S.; Geiseler, Charles J.; Greenfield, John A.; McNamara, Laura A.; Adams, Susan S.

Information visualization tools are being promoted to aid decision support. These tools assist in the analysis and comprehension of ambiguous and conflicting data sets. Formal evaluations are necessary to demonstrate the effectiveness of visualization tools, yet conducting these studies is difficult. Objective metrics that allow designers to compare the amount of work required for users to operate a particular interface are lacking. This in turn makes it difficult to compare workload across different interfaces, which is problematic for complicated information visualization and visual analytics packages. We believe that measures of working memory load can provide a more objective and consistent way of assessing visualizations and user interfaces across a range of applications. We present initial findings from a study using measures of working memory load to compare the usability of two graph representations. © 2011 Springer-Verlag.

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Lakkaraju, Kiran; Adams, Susan S.; Abbott, Robert G.; Forsythe, James C.

In this paper we performed analysis of speech communications in order to determine if we can differentiate between expert and novice teams based on communication patterns. Two pairs of experts and novices performed numerous test sessions on the E-2 Enhanced Deployable Readiness Trainer (EDRT) which is a medium-fidelity simulator of the Naval Flight Officer (NFO) stations positioned at bank end of the E-2 Hawkeye. Results indicate that experts and novices can be differentiated based on communication patterns. First, experts and novices differ significantly with regard to the frequency of utterances, with both expert teams making many fewer radio calls than both novice teams. Next, the semantic content of utterances was considered. Using both manual and automated speech-to-text conversion, the resulting text documents were compared. For 7 of 8 subjects, the two most similar subjects (using cosine-similarity of term vectors) were in the same category of expertise (novice/expert). This means that the semantic content of utterances by experts was more similar to other experts, than novices, and vice versa. Finally, using machine learning techniques we constructed a classifier that, given as input the text of the speech of a subject, could identify whether the individual was an expert or novice with a very low error rate. By looking at the parameters of the machine learning algorithm we were also able to identify terms that are strongly associated with novices and experts. © 2011 Springer-Verlag.

Matzen, Laura E.; Cole, Kerstan S.; Dornburg, Courtney S.; Bandlow, Alisa B.; Gieseler, Charles J.; McNamara, Laura A.; Adams, Susan S.

Abstract not provided.

Forsythe, James C.; Abbott, Robert G.; Adams, Susan S.; Hendrickson, Stacey M.; Trumbo, Michael C.; Haass, Michael J.

Abstract not provided.

Lakkaraju, Kiran L.; Adams, Susan S.; Abbott, Robert G.; Forsythe, James C.

Abstract not provided.

Bandlow, Alisa B.; Cole, Kerstan S.; McNamara, Laura A.; Dornburg, Courtney S.; Adams, Susan S.; Matzen, Laura E.; Gieseler, Charles J.

Abstract not provided.

Adams, Susan S.; Trumbo, Michael C.; Hendrickson, Stacey M.; Abbott, Robert G.; Haass, Michael J.

Abstract not provided.

Bandlow, Alisa B.; Matzen, Laura E.; Cole, Kerstan S.; Dornburg, Courtney S.; Gieseler, Charles J.; McNamara, Laura A.; Adams, Susan S.

Abstract not provided.

Adams, Susan S.; Basilico, Justin D.; Abbott, Robert G.

Training simulators have become increasingly popular tools for instructing humans on performance in complex environments. However, the question of how to provide individualized and scenario-specific assessment and feedback to students remains largely an open question. In this work, we follow-up on previous evaluations of the Automated Expert Modeling and Automated Student Evaluation (AEMASE) system, which automatically assesses student performance based on observed examples of good and bad performance in a given domain. The current study provides a rigorous empirical evaluation of the enhanced training effectiveness achievable with this technology. In particular, we found that students given feedback via the AEMASE-based debrief tool performed significantly better than students given only instructor feedback on two out of three domain-specific performance metrics.

Adams, Susan S.; Basilico, Justin D.; Abbott, Robert G.

Training simulators have become increasingly popular tools for instructing humans on performance in complex environments. However, the question of how to provide individualized and scenario-specific assessment and feedback to students remains largely an open question. To maximize training efficiency, new technologies are required that assist instructors in providing individually relevant instruction. Sandia National Laboratories has shown the feasibility of automated performance assessment tools, such as the Sandia-developed Automated Expert Modeling and Student Evaluation (AEMASE) software, through proof-of-concept demonstrations, a pilot study, and an experiment. In the pilot study, the AEMASE system, which automatically assesses student performance based on observed examples of good and bad performance in a given domain, achieved a high degree of agreement with a human grader (89%) in assessing tactical air engagement scenarios. In more recent work, we found that AEMASE achieved a high degree of agreement with human graders (83-99%) for three Navy E-2 domain-relevant performance metrics. The current study provides a rigorous empirical evaluation of the enhanced training effectiveness achievable with this technology. In particular, we assessed whether giving students feedback based on automated metrics would enhance training effectiveness and improve student performance. We trained two groups of employees (differentiated by type of feedback) on a Navy E-2 simulator and assessed their performance on three domain-specific performance metrics. We found that students given feedback via the AEMASE-based debrief tool performed significantly better than students given only instructor feedback on two out of three metrics. Future work will focus on extending these developments for automated assessment of teamwork.

Dornburg, Courtney S.; Adams, Susan S.; Hendrickson, Stacey M.; Davidson, George S.

An experiment was conducted comparing the effectiveness of individual versus group electronic brainstorming in order to address difficult, real world challenges. While industrial reliance on electronic communications has become ubiquitous, empirical and theoretical understanding of the bounds of its effectiveness have been limited. Previous research using short-term, laboratory experiments have engaged small groups of students in answering questions irrelevant to an industrial setting. The present experiment extends current findings beyond the laboratory to larger groups of real-world employees addressing organization-relevant challenges over the course of four days. Employees and contractors at a national security laboratory participated, either in a group setting or individually, in an electronic brainstorm to pose solutions to a 'wickedly' difficult problem. The data demonstrate that (for this design) individuals perform at least as well as groups in producing quantity of electronic ideas, regardless of brainstorming duration. However, when judged with respect to quality along three dimensions (originality, feasibility, and effectiveness), the individuals significantly (p<0.05) out-performed the group working together. When idea quality is used as the benchmark of success, these data indicate that work-relevant challenges are better solved by aggregating electronic individual responses, rather than electronically convening a group. This research suggests that industrial reliance upon electronic problem solving groups should be tempered, and large nominal groups might be the more appropriate vehicle for solving wicked corporate issues.

Adams, Susan S.; Hendrickson, Stacey M.; Dornburg, Courtney S.; Davidson, George S.

Abstract not provided.

Dornburg, Courtney S.; Adams, Susan S.; Forsythe, James C.; Davidson, George S.

The present paper explores group dynamics and electronic communication, two components of wicked problem solving that are inherent to the national security environment (as well as many other business environments). First, because there can be no ''right'' answer or solution without first having agreement about the definition of the problem and the social meaning of a ''right solution'', these problems (often) fundamentally relate to the social aspects of groups, an area with much empirical research and application still needed. Second, as computer networks have been increasingly used to conduct business with decreased costs, increased information accessibility, and rapid document, database, and message exchange, electronic communication enables a new form of problem solving group that has yet to be well understood, especially as it relates to solving wicked problems.

Adams, Susan S.; Davidson, George S.; Dornburg, Courtney S.; Forsythe, James C.

An experiment is proposed which will compare the effectiveness of individual versus group brainstorming in addressing difficult, real world challenges. Previous research into electronic brainstorming has largely been limited to laboratory experiments using small groups of students answering questions irrelevant to an industrial setting. The proposed experiment attempts to extend current findings to real-world employees and organization-relevant challenges. Our employees will brainstorm ideas over the course of several days, echoing the real-world scenario in an industrial setting. The methodology and hypotheses to be tested are presented along with two questions for the experimental brainstorming sessions. One question has been used in prior work and will allow calibration of the new results with existing work. The second question qualifies as a complicated, perhaps even wickedly hard, question, with relevance to modern management practices.

Davidson, George S.; Dornburg, Courtney S.; Adams, Susan S.; Hendrickson, Stacey M.; Bauer, Travis L.; Forsythe, James C.

An experiment was conducted comparing the effectiveness of individual versus group electronic brainstorming in order to address difficult, real world challenges. While industrial reliance on electronic communications has become ubiquitous, empirical and theoretical understanding of the bounds of its effectiveness have been limited. Previous research using short-term, laboratory experiments have engaged small groups of students in answering questions irrelevant to an industrial setting. The current experiment extends current findings beyond the laboratory to larger groups of real-world employees addressing organization-relevant challenges over the course of four days. Findings are twofold. First, the data demonstrate that (for this design) individuals perform at least as well as groups in producing quantity of electronic ideas, regardless of brainstorming duration. However, when judged with respect to quality along three dimensions (originality, feasibility, and effectiveness), the individuals significantly (p<0.05) out performed the group working together. The theoretical and applied (e.g., cost effectiveness) implications of this finding are discussed. Second, the current experiment yielded several viable solutions to the wickedly difficult problem that was posed.