Complex challenges across Sandia National Laboratories? (SNL) mission areas underscore the need for systems level thinking, resulting in a better understanding of the organizational work systems and environments in which our hardware and software will be used. SNL researchers have successfully used Activity Theory (AT) as a framework to clarify work systems, informing product design, delivery, acceptance, and use. To increase familiarity with AT, a working group assembled to select key resources on the topic and generate an annotated bibliography. The resources in this bibliography are arranged in six categories: 1) An introduction to AT; 2) Advanced readings in AT; 3) AT and human computer interaction (HCI); 4) Methodological resources for practitioners; 5) Case studies; and 6) Related frameworks that have been used to study work systems. This annotated bibliography is expected to improve the reader?s understanding of AT and enable more efficient and effective application of it.
Analysts develop a “no threat” bias with high false alarms. If only shown alarms for actual attacks, may never actually see an alarm. We see this in the laboratory, but not often studied in applied environments. (TSA is an exception.) In this work, near-operational paradigms are useful, but difficult to construct well. Pilot testing is critical before engaging time-limited professionals. Experimental control is difficult to balance with operational realism. Grounding near-operational experiments in basic research paradigms has both advantages and disadvantages. Despite shortcomings in our second experiment, we now have a platform for experimental investigations into the human element of physical security systems.
There is a great deal of debate concerning the benefits of working memory (WM) training and whether that training can transfer to other tasks. Although a consistent finding is that WM training programs elicit a short-term near-transfer effect (i.e., improvement in WM skills), results are inconsistent when considering persistence of such improvement and far transfer effects. In this study, we compared three groups of participants: a group that received WM training, a group that received training on how to use a mental imagery memory strategy, and a control group that received no training. Although the WM training group improved on the trained task, their posttraining performance on nontrained WM tasks did not differ from that of the other two groups. In addition, although the imagery training group’s performance on a recognition memory task increased after training, the WM training group’s performance on the task decreased after training. Participants’ descriptions of the strategies they used to remember the studied items indicated that WM training may lead people to adopt memory strategies that are less effective for other types of memory tasks. These results indicate that WM training may have unintended consequences for other types of memory performance.
Research, the manufacture of knowledge, is currently practiced largely as an “art,” not a “science.” Just as science (understanding) and technology (tools) have revolutionized the manufacture of other goods and services, it is natural, perhaps inevitable, that they will ultimately also be applied to the manufacture of knowledge. In this article, we present an emerging perspective on opportunities for such application, at three different levels of the research enterprise. At the cognitive science level of the individual researcher, opportunities include: overcoming idea fixation and sloppy thinking, and balancing divergent and convergent thinking. At the social network level of the research team, opportunities include: overcoming strong links and groupthink, and optimally distributing divergent and convergent thinking between individuals and teams. At the research ecosystem level of the research institution and the larger national and international community of researchers, opportunities include: overcoming performance fixation, overcoming narrow measures of research impact, and overcoming (or harnessing) existential/social stress.
Human performance has become a pertinent issue within cyber security. However, this research has been stymied by the limited availability of expert cyber security professionals. This is partly attributable to the ongoing workload faced by cyber security professionals, which is compound ed by the limited number of qualified personnel and turnover of personnel across organizations. Additionally, it is difficult to conduct research, and particularly, openly published research, due to the sensitivity inherent to cyber ope rations at most organizations. As an alternative, the current research has focused on data collection during cyber security training exercises. These events draw individuals with a range of knowledge and experience extending from seasoned professionals to recent college graduates to college students. The current paper describes research involving data collection at two separate cyber security exercises. This data collection involved multiple measures which included behavioral performance based on human - machine transactions and questionnaire - based assessments of cyber security experience.
Previously, the current authors (Hopkins et al. 2015) described research in which subjects provided a tool that facilitated their construction of a narrative account of events performed better in conducting cyber security forensic analysis. The narrative tool offered several distinct features. In the current paper, an analysis is reported that considered which features of the tool contributed to superior performance. This analysis revealed two features that accounted for a statistically significant portion of the variance in performance. The first feature provided a mechanism for subjects to identify suspected perpetrators of the crimes and their motives. The second feature involved the ability to create an annotated visuospatial diagram of clues regarding the crimes and their relationships to one another. Based on these results, guidance may be provided for the development of software tools meant to aid cyber security professionals in conducting forensic analysis.
Criminal forensic analysis involves examining a collection of clues to construct a plausible account of the events associated with a crime. In this paper, a study is presented that assessed whether software tools designed to encourage construction of narrative accounts would facilitate cyber forensic analysis. Compared to a baseline condition (i.e., spreadsheet with note-taking capabilities) and a visualization condition, subjects performed best when provided tools that emphasized established components of narratives. Specifically, features that encouraged subjects to identify suspected entities, and their activities and motivations proved beneficial. It is proposed that software tools developed to facilitate cyber forensic analysis and training of cyber security professionals incorporate techniques that facilitate a narrative account of events.
The Transportation Security Administration has a large workforce of Transportation Security Officers, most of whom perform interrogation of x-ray images at the passenger checkpoint. To date, TSOs on the x-ray have been limited to a 30-min session at a time, however, it is unclear where this limit originated. The current paper outlines methods for empirically determining if that 30-min duty cycle is optimal and if there are differences between individual TSOs. This work can inform scheduling TSOs at the checkpoint and can also inform whether TSOs should continue to be cross-trained (i.e., performing all 6 checkpoint duties) or whether specialization makes more sense.
Visual search data describe people’s performance on the common perceptual problem of identifying target objects in a complex scene. Technological advances in areas such as eye tracking now provide researchers with a wealth of data not previously available. The goal of this work is to support researchers in analyzing this complex and multimodal data and in developing new insights into visual search techniques. We discuss several methods drawn from the statistics and machine learning literature for integrating visual search data derived from multiple sources and performing exploratory data analysis. We ground our discussion in a specific task performed by officers at the Transportation Security Administration and consider the applicability, likely issues, and possible adaptations of several candidate analysis methods.
Numerous domains, ranging from medical diagnostics to intelligence analysis, involve visual search tasks in which people must find and identify specific items within large sets of imagery. These tasks rely heavily on human judgment, making fully automated systems infeasible in many cases. Researchers have investigated methods for combining human judgment with computational processing to increase the speed at which humans can triage large image sets. One such method is rapid serial visual presentation (RSVP), in which images are presented in rapid succession to a human viewer. While viewing the images and looking for targets of interest, the participant’s brain activity is recorded using electroencephalography (EEG). The EEG signals can be time-locked to the presentation of each image, producing event-related potentials (ERPs) that provide information about the brain’s response to those stimuli. The participants’ judgments about whether or not each set of images contained a target and the ERPs elicited by target and non-target images are used to identify subsets of images that merit close expert scrutiny [1]. Although the RSVP/EEG paradigm holds promise for helping professional visual searchers to triage imagery rapidly, it may be limited by the nature of the target items. Targets that do not vary a great deal in appearance are likely to elicit useable ERPs, but more variable targets may not. In the present study, we sought to extend the RSVP/EEG paradigm to the domain of aviation security screening, and in doing so to explore the limitations of the technique for different types of targets. Professional Transportation Security Officers (TSOs) viewed bag X-rays that were presented using an RSVP paradigm. The TSOs viewed bursts of images containing 50 segments of bag X-rays that were presented for 100 ms each. Following each burst of images, the TSOs indicated whether or not they thought there was a threat item in any of the images in that set. EEG was recorded during each burst of images and ERPs were calculated by time-locking the EEG signal to the presentation of images containing threats and matched images that were identical except for the presence of the threat item. Half of the threat items had a prototypical appearance and half did not. We found that the bag images containing threat items with a prototypical appearance reliably elicited a P300 ERP component, while those without a prototypical appearance did not. These findings have implications for the application of the RSVP/EEG technique to real-world visual search domains.
Sandia Labs has corporate, lab-wide efforts to enhance the research environment as well as improve physical space. However, these two efforts are usually done in isolation. The integration of physical space design with the nurturing of what we call psychosocial space can foster more efficient and effective creativity, innovation, collaboration, and performance. This paper presents a brief literature review on how academia and industry are studying the integration of physical and psychosocial space and focuses on the efforts that we, the authors, have made to improve the research environment in the Cyber Engineering Research Lab (CERL), home to Group 1460. Interviews with subject matter experts from Silicon Valley and the University of New Mexico plus changes to actual spaces in CERL provided us with six lessons learned when integrating physical and psychosocial space. We describe these six key takeaways in hopes that Sandia will see this area as an evolving research capability that Sandia can both contribute to and benefit from.
Within cyber security, the human element represents one of the greatest untapped opportunities for increasing the effectiveness of network defenses. However, there has been little research to understand the human dimension in cyber operations. To better understand the needs and priorities for research and development to address these issues, a workshop was conducted August 28-29, 2012 in Washington DC. A synthesis was developed that captured the key issues and associated research questions. Research and development needs were identified that fell into three parallel paths: (1) human factors analysis and scientific studies to establish foundational knowledge concerning factors underlying the performance of cyber defenders; (2) development of models that capture key processes that mediate interactions between defenders, users, adversaries and the public; and (3) development of a multi-purpose test environment for conducting controlled experiments that enables systems and human performance measurement. These research and development investments would transform cyber operations from an art to a science, enabling systems solutions to be engineered to address a range of situations. Organizations would be able to move beyond the current state where key decisions (e.g. personnel assignment) are made on a largely ad hoc basis to a state in which there exist institutionalized processes for assuring the right people are doing the right jobs in the right way. These developments lay the groundwork for emergence of a professional class of cyber defenders with defined roles and career progressions, with higher levels of personnel commitment and retention. Finally, the operational impact would be evident in improved performance, accompanied by a shift to a more proactive response in which defenders have the capacity to exert greater control over the cyber battlespace.