Publications

Abstract not provided.

Abstract not provided.

Abstract not provided.

Human performance has become a pertinent issue within cyber security. However, this research has been stymied by the limited availability of expert cyber security professionals. This is partly attributable to the ongoing workload faced by cyber security professionals, which is compound ed by the limited number of qualified personnel and turnover of personnel across organizations. Additionally, it is difficult to conduct research, and particularly, openly published research, due to the sensitivity inherent to cyber ope rations at most organizations. As an alternative, the current research has focused on data collection during cyber security training exercises. These events draw individuals with a range of knowledge and experience extending from seasoned professionals to recent college graduates to college students. The current paper describes research involving data collection at two separate cyber security exercises. This data collection involved multiple measures which included behavioral performance based on human - machine transactions and questionnaire - based assessments of cyber security experience.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

In cybersecurity forensics and incident response, the story of what has happened is the most important artifact yet the one least supported by tools and techniques. Existing tools focus on gathering and manipulating low-level data to allow an analyst to investigate exactly what happened on a host system or a network. Higher-level analysis is usually left to whatever ad hoc tools and techniques an individual may have developed. We discuss visual representations of narrative in the context of cybersecurity incidents with an eye toward multi-scale illustration of actions and actors. We envision that this representation could smoothly encompass individual packets on a wire at the lowest level and nation-state-level actors at the highest. We present progress to date, discuss the impact of technical risk on this project and highlight opportunities for future work.

Previously, the current authors (Hopkins et al. 2015) described research in which subjects provided a tool that facilitated their construction of a narrative account of events performed better in conducting cyber security forensic analysis. The narrative tool offered several distinct features. In the current paper, an analysis is reported that considered which features of the tool contributed to superior performance. This analysis revealed two features that accounted for a statistically significant portion of the variance in performance. The first feature provided a mechanism for subjects to identify suspected perpetrators of the crimes and their motives. The second feature involved the ability to create an annotated visuospatial diagram of clues regarding the crimes and their relationships to one another. Based on these results, guidance may be provided for the development of software tools meant to aid cyber security professionals in conducting forensic analysis.

Criminal forensic analysis involves examining a collection of clues to construct a plausible account of the events associated with a crime. In this paper, a study is presented that assessed whether software tools designed to encourage construction of narrative accounts would facilitate cyber forensic analysis. Compared to a baseline condition (i.e., spreadsheet with note-taking capabilities) and a visualization condition, subjects performed best when provided tools that emphasized established components of narratives. Specifically, features that encouraged subjects to identify suspected entities, and their activities and motivations proved beneficial. It is proposed that software tools developed to facilitate cyber forensic analysis and training of cyber security professionals incorporate techniques that facilitate a narrative account of events.

Electric distribution utilities, the companies that feed electricity to end users, are overseeing a technological transformation of their networks, installing sensors and other automated equipment, that are fundamentally changing the way the grid operates. These grid modernization efforts will allow utilities to incorporate some of the newer technology available to the home user – such as solar panels and electric cars – which will result in a bi-directional flow of energy and information. How will this new flow of information affect control room operations? How will the increased automation associated with smart grid technologies influence control room operators’ decisions? And how will changes in control room operations and operator decision making impact grid resilience? These questions have not been thoroughly studied, despite the enormous changes that are taking place. In this study, which involved collaborating with utility companies in the state of Vermont, the authors proposed to advance the science of control-room decision making by understanding the impact of distribution grid modernization on operator performance. Distribution control room operators were interviewed to understand daily tasks and decisions and to gain an understanding of how these impending changes will impact control room operations. Situation awareness was found to be a major contributor to successful control room operations. However, the impact of growing levels of automation due to smart grid technology on operators’ situation awareness is not well understood. Future work includes performing a naturalistic field study in which operator situation awareness will be measured in real-time during normal operations and correlated with the technological changes that are underway. The results of this future study will inform tools and strategies that will help system operators adapt to a changing grid, respond to critical incidents and maintain critical performance skills.

The impact of automation on human performance has been studied by human factors researchers for over 35 years. One unresolved facet of this research is measurement of the level of automation across and within engineered systems. Repeatable methods of observing, measuring and documenting the level of automation are critical to the creation and validation of generalized theories of automation's impact on the reliability and resilience of human-in-the-loop systems. Numerous qualitative scales for measuring automation have been proposed. However these methods require subjective assessments based on the researcher's knowledge and experience, or through expert knowledge elicitation involving highly experienced individuals from each work domain. More recently, quantitative scales have been proposed, but have yet to be widely adopted, likely due to the difficulty associated with obtaining a sufficient number of empirical measurements from each system component. Our research suggests the need for a quantitative method that enables rapid measurement of a system's level of automation, is applicable across domains, and can be used by human factors practitioners in field studies or by system engineers as part of their technical planning processes. In this paper we present our research methodology and early research results from studies of electricity grid distribution control rooms. Using a system analysis approach based on quantitative measures of level of automation, we provide an illustrative analysis of select grid modernization efforts. This measure of the level of automation can be displayed as either a static, historical view of the system's automation dynamics (the dynamic interplay between human and automation required to maintain system performance) or it can be incorporated into real-time visualization systems already present in control rooms.

There is a great need for creating cohesive, expert cybersecurity incident response teams and training them effectively. This paper discusses new methodologies for measuring and understanding expert and novice differences within a cybersecurity environment to bolster training, selection, and teaming. This methodology for baselining and characterizing individuals and teams relies on relating eye tracking gaze patterns to psychological assessments, human-machine transaction monitoring, and electroencephalography data that are collected during participation in the game-based training platform Tracer FIRE. We discuss preliminary findings from two pilot studies using novice and professional teams.

Abstract not provided.

Adaptive Thinking has been defined here as the capacity to recognize when a course of action that may have previously been effective is no longer effective and there is need to adjust strategy. Research was undertaken with human test subjects to identify the factors that contribute to adaptive thinking. It was discovered that those most effective in settings that call for adaptive thinking tend to possess a superior capacity to quickly and effectively generate possible courses of action, as measured using the Category Generation test. Software developed for this research has been applied to develop capabilities enabling analysts to identify crucial factors that are predictive of outcomes in fore-on-force simulation exercises.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Within cyber security, the human element represents one of the greatest untapped opportunities for increasing the effectiveness of network defenses. However, there has been little research to understand the human dimension in cyber operations. To better understand the needs and priorities for research and development to address these issues, a workshop was conducted August 28-29, 2012 in Washington DC. A synthesis was developed that captured the key issues and associated research questions. Research and development needs were identified that fell into three parallel paths: (1) human factors analysis and scientific studies to establish foundational knowledge concerning factors underlying the performance of cyber defenders; (2) development of models that capture key processes that mediate interactions between defenders, users, adversaries and the public; and (3) development of a multi-purpose test environment for conducting controlled experiments that enables systems and human performance measurement. These research and development investments would transform cyber operations from an art to a science, enabling systems solutions to be engineered to address a range of situations. Organizations would be able to move beyond the current state where key decisions (e.g. personnel assignment) are made on a largely ad hoc basis to a state in which there exist institutionalized processes for assuring the right people are doing the right jobs in the right way. These developments lay the groundwork for emergence of a professional class of cyber defenders with defined roles and career progressions, with higher levels of personnel commitment and retention. Finally, the operational impact would be evident in improved performance, accompanied by a shift to a more proactive response in which defenders have the capacity to exert greater control over the cyber battlespace.

Abstract not provided.

This report summarizes research conducted through the Sandia National Laboratories Enhanced Training for Cyber Situational Awareness in Red Versus Blue Team Exercises Laboratory Directed Research and Development project. The objective of this project was to advance scientific understanding concerning how to best structure training for cyber defenders. Two modes of training were considered. The baseline training condition (Tool-Based training) was based on current practices where classroom instruction focuses on the functions of a software tool with various exercises in which students apply those functions. In the second training condition (Narrative-Based training), classroom instruction addressed software functions, but in the context of adversary tactics and techniques. It was hypothesized that students receiving narrative-based training would gain a deeper conceptual understanding of the software tools and this would be reflected in better performance within a red versus blue team exercise.

Abstract not provided.

Abstract not provided.

Many enterprises are becoming increasingly data-driven. For example, empirically collected data about customer behavior offers an alternative to more traditional, synthetic techniques such as surveys, focus groups, and subject-matter experts. In contrast, recordings of tactical training exercises for the US military are not broadly archived or available for analysis. There may be great opportunity for military training and planning to use analogous techniques, where tactical scenarios are systematically recorded, indexed, and archived. Such a system would provide information for all levels of analysis, including establishing benchmarks for individual performance, evaluating the relevance and impact of training protocols, and assessing the utility of proposed systems and conops. However, such a system also offers many challenges and risks, such as cost, security, privacy, and end-user accessibility. This paper examines the possible benefits and risks of such a system with some emphasis on our recent research to address end-user accessibility.

Abstract not provided.

In this paper we performed analysis of speech communications in order to determine if we can differentiate between expert and novice teams based on communication patterns. Two pairs of experts and novices performed numerous test sessions on the E-2 Enhanced Deployable Readiness Trainer (EDRT) which is a medium-fidelity simulator of the Naval Flight Officer (NFO) stations positioned at bank end of the E-2 Hawkeye. Results indicate that experts and novices can be differentiated based on communication patterns. First, experts and novices differ significantly with regard to the frequency of utterances, with both expert teams making many fewer radio calls than both novice teams. Next, the semantic content of utterances was considered. Using both manual and automated speech-to-text conversion, the resulting text documents were compared. For 7 of 8 subjects, the two most similar subjects (using cosine-similarity of term vectors) were in the same category of expertise (novice/expert). This means that the semantic content of utterances by experts was more similar to other experts, than novices, and vice versa. Finally, using machine learning techniques we constructed a classifier that, given as input the text of the speech of a subject, could identify whether the individual was an expert or novice with a very low error rate. By looking at the parameters of the machine learning algorithm we were also able to identify terms that are strongly associated with novices and experts. © 2011 Springer-Verlag.

Abstract not provided.

Abstract not provided.

Abstract not provided.

While individual neurons function at relatively low firing rates, naturally-occurring nervous systems not only surpass manmade systems in computing power, but accomplish this feat using relatively little energy. It is asserted that the next major breakthrough in computing power will be achieved through application of neuromorphic approaches that mimic the mechanisms by which neural systems integrate and store massive quantities of data for real-time decision making. The proposed LDRD provides a conceptual foundation for SNL to make unique advances toward exascale computing. First, a team consisting of experts from the HPC, MESA, cognitive and biological sciences and nanotechnology domains will be coordinated to conduct an exercise with the outcome being a concept for applying neuromorphic computing to achieve exascale computing. It is anticipated that this concept will involve innovative extension and integration of SNL capabilities in MicroFab, material sciences, high-performance computing, and modeling and simulation of neural processes/systems.

Training simulators have become increasingly popular tools for instructing humans on performance in complex environments. However, the question of how to provide individualized and scenario-specific assessment and feedback to students remains largely an open question. In this work, we follow-up on previous evaluations of the Automated Expert Modeling and Automated Student Evaluation (AEMASE) system, which automatically assesses student performance based on observed examples of good and bad performance in a given domain. The current study provides an empirical evaluation of the enhanced training effectiveness achievable with this technology. In particular, we found that students given feedback via the AEMASE-based debrief tool performed significantly better than students given only instructor feedback.

This report summarizes accomplishments of a three-year project focused on developing technical capabilities for measuring and modeling neuronal processes at the nanoscale. It was successfully demonstrated that nanoprobes could be engineered that were biocompatible, and could be biofunctionalized, that responded within the range of voltages typically associated with a neuronal action potential. Furthermore, the Xyce parallel circuit simulator was employed and models incorporated for simulating the ion channel and cable properties of neuronal membranes. The ultimate objective of the project had been to employ nanoprobes in vivo, with the nematode C elegans, and derive a simulation based on the resulting data. Techniques were developed allowing the nanoprobes to be injected into the nematode and the neuronal response recorded. To the authors's knowledge, this is the first occasion in which nanoparticles have been successfully employed as probes for recording neuronal response in an in vivo animal experimental protocol.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

The present paper explores group dynamics and electronic communication, two components of wicked problem solving that are inherent to the national security environment (as well as many other business environments). First, because there can be no ''right'' answer or solution without first having agreement about the definition of the problem and the social meaning of a ''right solution'', these problems (often) fundamentally relate to the social aspects of groups, an area with much empirical research and application still needed. Second, as computer networks have been increasingly used to conduct business with decreased costs, increased information accessibility, and rapid document, database, and message exchange, electronic communication enables a new form of problem solving group that has yet to be well understood, especially as it relates to solving wicked problems.

An experiment is proposed which will compare the effectiveness of individual versus group brainstorming in addressing difficult, real world challenges. Previous research into electronic brainstorming has largely been limited to laboratory experiments using small groups of students answering questions irrelevant to an industrial setting. The proposed experiment attempts to extend current findings to real-world employees and organization-relevant challenges. Our employees will brainstorm ideas over the course of several days, echoing the real-world scenario in an industrial setting. The methodology and hypotheses to be tested are presented along with two questions for the experimental brainstorming sessions. One question has been used in prior work and will allow calibration of the new results with existing work. The second question qualifies as a complicated, perhaps even wickedly hard, question, with relevance to modern management practices.

An experiment was conducted comparing the effectiveness of individual versus group electronic brainstorming in order to address difficult, real world challenges. While industrial reliance on electronic communications has become ubiquitous, empirical and theoretical understanding of the bounds of its effectiveness have been limited. Previous research using short-term, laboratory experiments have engaged small groups of students in answering questions irrelevant to an industrial setting. The current experiment extends current findings beyond the laboratory to larger groups of real-world employees addressing organization-relevant challenges over the course of four days. Findings are twofold. First, the data demonstrate that (for this design) individuals perform at least as well as groups in producing quantity of electronic ideas, regardless of brainstorming duration. However, when judged with respect to quality along three dimensions (originality, feasibility, and effectiveness), the individuals significantly (p<0.05) out performed the group working together. The theoretical and applied (e.g., cost effectiveness) implications of this finding are discussed. Second, the current experiment yielded several viable solutions to the wickedly difficult problem that was posed.

A web-based brainstorm was conducted in the summer of 2007 within the Sandia Restricted Network. This brainstorming experiment was modeled around the 'yellow sticky' brainstorms that are used in many face-to-face meetings at Sandia National Laboratories. This document discusses the implementation and makes suggestions for future implementations.

Abstract not provided.

Abstract not provided.

This paper highlights key topic areas to be discussed the authors in a panel format during the Augmented Cognition thematic area paper session: 'Augmented Cognition Lessons Learned and Future Directions for Enabling 'Anyone, Anytime, Anywhere' Applications'. The term 'killer app' has been part of the vernacular in the commercial computer software and electronic devices industry to refer to breakthrough technologies [2]. A 'killer app' generally emerges with the development of related technologies that extends over some time and involves numerous variations on a basic concept. Hypotheses may be offered with respect to the conditions that will be needed to enable a similar situation with augmented cognition technologies. This paper and resulting panel session will address the numerous concepts that have emerged from the augmented cognition field to date and postulate how and when this field's first 'killer app' may emerge (e.g., 5, 10, 15, or more years from now).

Abstract not provided.

Abstract not provided.

This paper highlights key topic areas to be discussed the authors in a panel format during the Augmented Cognition thematic area paper session: 'Augmented Cognition Lessons Learned and Future Directions for Enabling 'Anyone, Anytime, Anywhere' Applications'. The term 'killer app' has been part of the vernacular in the commercial computer software and electronic devices industry to refer to breakthrough technologies [2]. A 'killer app' generally emerges with the development of related technologies that extends over some time and involves numerous variations on a basic concept. Hypotheses may be offered with respect to the conditions that will be needed to enable a similar situation with augmented cognition technologies. This paper and resulting panel session will address the numerous concepts that have emerged from the augmented cognition field to date and postulate how and when this field's first 'killer app' may emerge (e.g., 5, 10, 15, or more years from now).

Abstract not provided.

An ongoing program of research and development is utilizing nanomaterials as a basis for observing and measuring neurophysiological processes. Work commencing in fiscal year 2007 will focus on expanding current capabilities to create nanoelectrode arrays that will allow nanoscale measurement of the activity of 10's to 100's of neurons. This development is a vital step in gaining scientific insights concerning network properties associated with neural representations and processes. Specifically, attention will be focused the representation of memory in the hippocampus, for which extensive research has been conducted using laboratory rats. This report summarizes background research providing a foundation for work planned for fiscal year 2007 and beyond. In particular, the neuroanatomy and neurophysiology of the hippocampus is described. Additionally, several programs of research are described that have addressed the relationship between neurophysiological processes and behavioral measures of memory performance. These studies provide insight into methodological and analytic approaches for studying the representation of memory processes in the hippocampus. The objective of this report is to document relevant literature in a reference document that will support future research in this area.

Abstract not provided.

Variables influencing inferences about a stranger's goal during an unsolicited social interaction were explored. Experiment 1 developed a procedure for identifying cues. Experiments 2 and 3 assessed the relative importance of various cues (space, time, characteristics of oneself, characteristics of the stranger, and the stranger's behavior) for goal judgments. Results indicated that situational context cues informed goal judgments in ways that were consistent with diagnosticity ratings and typicality ratings of those cues. Stranger characteristics and stranger behaviors affected goal judgments more than would be expected from these quantitative measures of their informativeness. Nonetheless, the results are consistent with a mental model view that assumes perceivers monitor situational cues present during interactions and that goal inferences are guided by the informativeness of these cues.

A classification system is developed to identify driving situations from labeled examples of previous occurrences. The purpose of the classifier is to provide physical context to a separate system that mitigates unnecessary distractions, allowing the driver to maintain focus during periods of high difficulty. While watching videos of driving, we asked different users to indicate their perceptions of the current situation. We then trained a classifier to emulate the human recognition of driving situations using the Sandia Cognitive Framework. In unstructured conditions, such as driving in urban areas and the German autobahn, the classifier was able to correctly predict human perceptions of driving situations over 95% of the time. This paper focuses on the learning algorithms used to train the driving-situation classifier. Future work will reduce the human efforts needed to train the system. © 2005 IEEE.

This report documents work undertaken to endow the cognitive framework currently under development at Sandia National Laboratories with a human-like memory for specific life episodes. Capabilities have been demonstrated within the context of three separate problem areas. The first year of the project developed a capability whereby simulated robots were able to utilize a record of shared experience to perform surveillance of a building to detect a source of smoke. The second year focused on simulations of social interactions providing a queriable record of interactions such that a time series of events could be constructed and reconstructed. The third year addressed tools to promote desktop productivity, creating a capability to query episodic logs in real time allowing the model of a user to build on itself based on observations of the user's behavior.

This project is being conducted by Sandia National Laboratories in support of the DARPA Augmented Cognition program. Work commenced in April of 2002. The objective for the DARPA program is to 'extend, by an order of magnitude or more, the information management capacity of the human-computer warfighter.' Initially, emphasis has been placed on detection of an operator's cognitive state so that systems may adapt accordingly (e.g., adjust information throughput to the operator in response to workload). Work conducted by Sandia focuses on development of technologies to infer an operator's ongoing cognitive processes, with specific emphasis on detecting discrepancies between machine state and an operator's ongoing interpretation of events.

This report documents an exploratory FY 00 LDRD project that sought to demonstrate the first steps toward a realistic computational representation of the variability encountered in individual human behavior. Realism, as conceptualized in this project, required that the human representation address the underlying psychological, cultural, physiological, and environmental stressors. The present report outlines the researchers' approach to representing cognitive, cultural, and physiological variability of an individual in an ambiguous situation while faced with a high-consequence decision that would greatly impact subsequent events. The present project was framed around a sensor-shooter scenario as a soldier interacts with an unexpected target (two young Iraqi girls). A software model of the ''Sensor Shooter'' scenario from Desert Storm was developed in which the framework consisted of a computational instantiation of Recognition Primed Decision Making in the context of a Naturalistic Decision Making model [1]. Recognition Primed Decision Making was augmented with an underlying foundation based on our current understanding of human neurophysiology and its relationship to human cognitive processes. While the Gulf War scenario that constitutes the framework for the Sensor Shooter prototype is highly specific, the human decision architecture and the subsequent simulation are applicable to other problems similar in concept, intensity, and degree of uncertainty. The goal was to provide initial steps toward a computational representation of human variability in cultural, cognitive, and physiological state in order to attain a better understanding of the full depth of human decision-making processes in the context of ambiguity, novelty, and heightened arousal.

The authors describe a naturalistic behavioral model for the simulation of small unit combat. This model, Klein's recognition-primed decision making (RPD) model, is driven by situational awareness rather than a rational process of selecting from a set of action options. They argue that simulated combatants modeled with RPD will have more flexible and realistic responses to a broad range of small-scale combat scenarios. Furthermore, they note that the predictability of a simulation using an RPD framework can be easily controlled to provide multiple evaluations of a given combat scenario. Finally, they discuss computational issues for building an RPD-based behavior engine for fully automated combatants in small conflict scenarios, which are being investigated within Sandia's Next Generation Site Security project.

Despite extensive safety analysis and application of safety measures, there is a frequent lament, "Why do we continue to have accidents?" Two breakdowns are prevalent in risk management and prevention. First, accidents result from human actions that engineers, analysts and management never envisioned and second, controls, intended to preclude/mitigate accident sequences, prove inadequate. This paper addresses the first breakdown, the inability to anticipate scenarios involving human action/inaction. The failure of controls has been addressed in a previous publication (Forsythe and Grose, 1998). Specifically, this paper presents an approach referred to as "surety." The objective of this approach is to provide high levels of assurance in situations where potential system failure paths cannot be fully characterized. With regard to human elements of complex systems, traditional approaches to human reliability are not sufficient to attain surety. Consequently, an Organic Model has been developed to account for the organic properties exhibited by engineered systems that result from human involvement in those systems.