Publications

Abstract not provided.

This report summarizes findings from a small, mixed-method research study examining industry perspectives on the potential for new forms of automation to invigorate the concentrating solar power (CSP) industry. In Fall 2021, the Solar Energy Technologies Office (SETO) of the United States Department of Energy (DOE) funded Sandia National Laboratories to elicit industry stakeholder perspectives on the potential role of automated systems in CSP operations. We interviewed eleven CSP professionals from five countries, using a combination of structured and open comment response modes. Respondents indicated a preference for automated systems that support heliostat manufacturing and installation, calibration, and responsiveness to shifting weather conditions. This pilot study demonstrates the importance of engaging industry stakeholders in discussions of technology research and development, to promote adoptable, useful innovation.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

This three-year Laboratory Directed Research and Development (LDRD) project aimed at developing a developed prototype data collection system and analysis techniques to enable the measurement and analysis of user-driven dynamic workflows. Over 3 years, our team developed software, algorithms, and analysis technique to explore the feasibility of capturing and automatically associating eye tracking data with geospatial content, in a user-directed, dynamic visual search task. Although this was a small LDRD, we demonstrated the feasibility of automatically capturing, associating, and expressing gaze events in terms of geospatial image coordinates, even as the human "analyst" is given complete freedom to manipulate the stimulus image during a visual search task. This report describes the problem under examination, our approach, the techniques and software we developed, key achievements, ideas that did not work as we had hoped, and unsolved problems we hope to tackle in future projects.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

With the rise of electronic and high-dimensional data, new and innovative feature detection and statistical methods are required to perform accurate and meaningful statistical analysis of these datasets that provide unique statistical challenges. In the area of feature detection, much of the recent feature detection research in the computer vision community has focused on deep learning methods, which require large amounts of labeled training data. However, in many application areas, training data is very limited and often difficult to obtain. We develop methods for fast, unsupervised, precise feature detection for video data based on optical flows, edge detection, and clustering methods. We also use pretrained neural networks and interpretable linear models to extract features using very limited training data. In the area of statistics, while high-dimensional data analysis has been a main focus of recent statistical methodological research, much focus has been on populations of high-dimensional vectors, rather than populations of high-dimensional tensors, which are three- dimensional arrays that can be used to model dependent images, such as images taken of the same person or ripped video frames. Our feature detection method is a non-model-based method that fusses information from dense optical flow, raw image pixels, and frame differences to generate detections. Our hypothesis testing methods are based on the assumption that dependent images are concatenated into a tensor that follows a tensor normal distribution, and from this assumption, we derive likelihood-ratio, score, and regression-based tests for one- and multiple-sample testing problems. Our methods will be illustrated on simulated and real datasets. We conclude this report with comments on the relationship between feature detection and hypothesis testing methods. Acknowledgements This work was funded by the Sandia National Laboratories Laboratory Directed Research and Development (LDRD) pro- gram.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Even as remote sensing technology has advanced in leaps and bounds over the past decade∗the remote sensing community lacks interfaces and interaction models that facilitate effective human operation of our sensor platforms. Interfaces that make great sense to electrical engineers and flight test crews can be anxiety-inducing to operational users who lack professional experience in the design and testing of sophisticated remote sensing platforms. In this paper, we reflect on an 18-month collaboration which our Sandia National Laboratory research team partnered with an industry software team to identify and fix critical issues in a widely-used sensor interface. Drawing on basic principles from cognitive and perceptual psychology and interaction design, we provide simple, easily learned guidance for minimizing common barriers to system learnability, memorability, and user engagement.

In this paper, we address the needed components to create usable engineering and operational user interfaces (UIs) for airborne Synthetic Aperture Radar (SAR) systems. As airborne SAR technology gains wider acceptance in the remote sensing and Intelligence, Surveillance, and Reconnaissance (ISR) communities, the need for effective and appropriate UIs to command and control these sensors has also increased. However, despite the growing demand for SAR in operational environments, the technology still faces an adoption roadblock, in large part due to the lack of effective UIs. It is common to find operational interfaces that have barely grown beyond the disparate tools engineers and technologists developed to demonstrate an initial concept or system. While sensor usability and utility are common requirements to engineers and operators, their objectives for interacting with the sensor are different. As such, the amount and type of information presented ought to be tailored to the specific application.

Researchers at Sandia National Laboratories in Albuquerque, New Mexico, are engaged in the empirical study of human-information interaction in high-consequence national security environments. This focus emerged from our longstanding interactions with military and civilian intelligence analysts working across a broad array of domains, from signals intelligence to cybersecurity to geospatial imagery analysis. In this paper, we discuss how several years’ of work with Synthetic Aperture Radar (SAR) imagery analysts revealed the limitations of eye tracking systems for capturing gaze events in the dynamic, user-driven problem-solving strategies characteristic of geospatial analytic workflows. We also explain the need for eye tracking systems capable of supporting inductive study of dynamic, user-driven problem-solving strategies characteristic of geospatial analytic workflows. We then discuss an ongoing project in which we are leveraging some of the unique properties of SAR image products to develop a prototype eyetracking data collection and analysis system that will support inductive studies of visual workflows in SAR image analysis environments.

Abstract not provided.

Abstract not provided.

In the context of text-based analysis workflows, we propose that an effective analytic tool facilitates triage by a) enabling users to identify and set aside irrelevant content (i.e., reduce the complexity of information in a dataset) and b) develop a working mental model of which items are most relevant to the question at hand. This LDRD funded research developed a dataset that is enabling this team to evaluate propose normalized compression distance (NCD) as a task, user, and context-insensitive measure of categorization outcomes (Shannon entropy is reduced as order is imposed). Effective analytics tools help people impose order, reducing complexity in measurable ways. Our concept and research was documented in a paper accepted to the ACM conference Beyond Time and Error: Novel Methods in Information Visualization Evaluation , part of the IEEE VisWeek Conference, Baltimore, MD, October 16-21, 2016. The paper is included as an appendix to this report.

Abstract not provided.

Abstract not provided.

Abstract not provided.

The evolution of exquisitely sensitive Synthetic Aperture Radar (SAR) systems is positioning this technology for use in time-critical environments, such as search-and-rescue missions and improvised explosive device (IED) detection. SAR systems should be playing a keystone role in the United States' Intelligence, Surveillance, and Reconnaissance activities. Yet many in the SAR community see missed opportunities for incorporating SAR into existing remote sensing data collection and analysis challenges. Drawing on several years' of field research with SAR engineering and operational teams, this paper examines the human and organizational factors that mitigate against the adoption and use of SAR for tactical ISR and operational support. We suggest that SAR has a design problem, and that context-sensitive, human and organizational design frameworks are required if the community is to realize SAR's tactical potential.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Researchers at Sandia National Laboratories are integrating qualitative and quantitative methods from anthropology, human factors and cognitive psychology in the study of military and civilian intelligence analyst workflows in the United States’ national security community. Researchers who study human work processes often use qualitative theory and methods, including grounded theory, cognitive work analysis, and ethnography, to generate rich descriptive models of human behavior in context. In contrast, experimental psychologists typically do not receive training in qualitative induction, nor are they likely to practice ethnographic methods in their work, since experimental psychology tends to emphasize generalizability and quantitative hypothesis testing over qualitative description. However, qualitative frameworks and methods from anthropology, sociology, and human factors can play an important role in enhancing the ecological validity of experimental research designs.

Vision is one of the dominant human senses and most human-computer interfaces rely heavily on the capabilities of the human visual system. An enormous amount of effort is devoted to finding ways to visualize information so that humans can understand and make sense of it. By studying how professionals engage in these visual search tasks, we can develop insights into their cognitive processes and the influence of experience on those processes. This can advance our understanding of visual cognition in addition to providing information that can be applied to designing improved data visualizations or training new analysts. In this study, we investigated the role of expertise on performance in a Synthetic Aperture Radar (SAR) target detection task. SAR imagery differs substantially from optical imagery, making it a useful domain for investigating expert-novice differences. The participants in this study included professional SAR imagery analysts, radar engineers with experience working with SAR imagery, and novices who had little or no prior exposure to SAR imagery. Participants from all three groups completed a domain-specific visual search task in which they searched for targets within pairs of SAR images. They also completed a battery of domain-general visual search and cognitive tasks that measured factors such as mental rotation ability, spatial working memory, and useful field of view. The results revealed marked differences between the professional imagery analysts and the other groups, both for the domain-specific task and for some domain-general tasks. These results indicate that experience with visual search in non-optical imagery can influence performance on other domains.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

The purpose of the current study was to analyze the work of imagery analysts associated with Sagebrush, a Synthetic Aperture Radar (SAR) imaging system, using an adapted version of cognitive work analysis (CWA). This was achieved by conducting a work domain analysis (WDA) for the system under consideration. Another purpose of this study was to describe how we adapted the WDA framework to include a sequential component and a means to explicitly represent relationships between components. Lastly, we present a simplified work domain representation that we have found effective in communicating the importance of analysts' adaptive strategies to inform the research strategies of computational science researchers who want to develop useful algorithms, but who have little or no familiarity with sensor data analysis work. © 2014 Springer International Publishing.

Imagery analysts are given the difficult task of determining, post-hoc, if particular events of importance had occurred, employing Synthetic Aperture Radar (SAR) images, written reports and PowerPoint presentations to make their decision. We were asked to evaluate the current system analysis process and make recommendations for a future temporal geospatial analysis prototype that is envisioned to allow analysts to quickly search for temporal and spatial relationships between image-derived features. As such, we conducted a Hierarchical task analysis (HTA; [3], [6]) to understand the analysts' tasks and subtasks. We also implemented a timeline analysis and workload assessment [4] to better understand which tasks were the most time-consuming and perceived as the most effortful. Our results gave the team clear recommendations and requirements for a prototype. © 2014 Springer International Publishing.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Information visualization tools are being promoted to aid decision support. These tools assist in the analysis and comprehension of ambiguous and conflicting data sets. Formal evaluations are necessary to demonstrate the effectiveness of visualization tools, yet conducting these studies is difficult. Objective metrics that allow designers to compare the amount of work required for users to operate a particular interface are lacking. This in turn makes it difficult to compare workload across different interfaces, which is problematic for complicated information visualization and visual analytics packages. We believe that measures of working memory load can provide a more objective and consistent way of assessing visualizations and user interfaces across a range of applications. We present initial findings from a study using measures of working memory load to compare the usability of two graph representations. © 2011 Springer-Verlag.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

The title of this paper, Why Models Don't Forecast, has a deceptively simple answer: models don't forecast because people forecast. Yet this statement has significant implications for computational social modeling and simulation in national security decision making. Specifically, it points to the need for robust approaches to the problem of how people and organizations develop, deploy, and use computational modeling and simulation technologies. In the next twenty or so pages, I argue that the challenge of evaluating computational social modeling and simulation technologies extends far beyond verification and validation, and should include the relationship between a simulation technology and the people and organizations using it. This challenge of evaluation is not just one of usability and usefulness for technologies, but extends to the assessment of how new modeling and simulation technologies shape human and organizational judgment. The robust and systematic evaluation of organizational decision making processes, and the role of computational modeling and simulation technologies therein, is a critical problem for the organizations who promote, fund, develop, and seek to use computational social science tools, methods, and techniques in high-consequence decision making.

Abstract not provided.

Abstract not provided.

Sandia National Laboratories is investing in projects that aim to develop computational modeling and simulation applications that explore human cognitive and social phenomena. While some of these modeling and simulation projects are explicitly research oriented, others are intended to support or provide insight for people involved in high consequence decision-making. This raises the issue of how to evaluate computational modeling and simulation applications in both research and applied settings where human behavior is the focus of the model: when is a simulation 'good enough' for the goals its designers want to achieve? In this report, we discuss two years' worth of review and assessment of the ASC program's approach to computational model verification and validation, uncertainty quantification, and decision making. We present a framework that extends the principles of the ASC approach into the area of computational social and cognitive modeling and simulation. In doing so, we argue that the potential for evaluation is a function of how the modeling and simulation software will be used in a particular setting. In making this argument, we move from strict, engineering and physics oriented approaches to V&V to a broader project of model evaluation, which asserts that the systematic, rigorous, and transparent accumulation of evidence about a model's performance under conditions of uncertainty is a reasonable and necessary goal for model evaluation, regardless of discipline. How to achieve the accumulation of evidence in areas outside physics and engineering is a significant research challenge, but one that requires addressing as modeling and simulation tools move out of research laboratories and into the hands of decision makers. This report provides an assessment of our thinking on ASC Verification and Validation, and argues for further extending V&V research in the physical and engineering sciences toward a broader program of model evaluation in situations of high consequence decision-making.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Since 1998, the Department of Energy/NNSA National Laboratories have invested millions in strategies for assessing the credibility of computational science and engineering (CSE) models used in high consequence decision making. The answer? There is no answer. There's a process--and a lot of politics. The importance of model evaluation (verification, validation, uncertainty quantification, and assessment) increases in direct proportion to the significance of the model as input to a decision. Other fields, including computational social science, can learn from the experience of the national laboratories. Some implications for evaluating 'low cognition agents'. Epistemology considers the question, How do we know what we [think we] know? What makes Western science special in producing reliable, predictive knowledge about the world? V&V takes epistemology out of the realm of thought and puts it into practice. What is the role of modeling and simulation in the production of reliable, credible scientific knowledge about the world? What steps, investments, practices do I pursue to convince myself that the model I have developed is producing credible knowledge?

Abstract not provided.

This white paper represents a summary of work intended to lay the foundation for development of a climatological/agent model of climate-induced conflict. The paper combines several loosely-coupled efforts and is the final report for a four-month late-start Laboratory Directed Research and Development (LDRD) project funded by the Advanced Concepts Group (ACG). The project involved contributions by many participants having diverse areas of expertise, with the common goal of learning how to tie together the physical and human causes and consequences of climate change. We performed a review of relevant literature on conflict arising from environmental scarcity. Rather than simply reviewing the previous work, we actively collected data from the referenced sources, reproduced some of the work, and explored alternative models. We used the unfolding crisis in Darfur (western Sudan) as a case study of conflict related to or triggered by climate change, and as an exercise for developing a preliminary concept map. We also outlined a plan for implementing agents in a climate model and defined a logical progression toward the ultimate goal of running both types of models simultaneously in a two-way feedback mode, where the behavior of agents influences the climate and climate change affects the agents. Finally, we offer some ''lessons learned'' in attempting to keep a diverse and geographically dispersed group working together by using Web-based collaborative tools.

Social and ecological scientists emphasize that effective natural resource management depends in part on understanding the dynamic relationship between the physical and non-physical process associated with resource consumption. In this case, the physical processes include hydrological, climatological and ecological dynamics, and the non-physical process include social, economic and cultural dynamics among humans who do the resource consumption. This project represents a case study aimed at modeling coupled social and physical processes in a single decision support system. In central New Mexico, individual land use decisions over the past five decades have resulted in the gradual transformation of the Middle Rio Grande Valley from a primarily rural agricultural landscape to a largely urban one. In the arid southwestern U.S., the aggregate impact of individual decisions about land use is uniquely important to understand, because scarce hydrological resources will likely limit the viability of resulting growth and development trajectories. This decision support tool is intended to help planners in the area look forward in their efforts to create a collectively defined 'desired' social landscape in the Middle Rio Grande. Our research question explored the ways in which socio-cultural values impact decisions regarding that landscape and associated land use. Because of the constraints hydrological resources place on land use, we first assumed that water use, as embodied in water rights, was a reasonable surrogate for land use. We thought that modeling the movement of water rights over time and across water source types (surface and ground) would provide planners with insight into the possibilities for certain types of decisions regarding social landscapes, and the impact those same decisions would have on those landscapes. We found that water rights transfer data in New Mexico is too incomplete and inaccurate to use as the basis for the model. Furthermore, because of its lack of accuracy and completeness, water rights ownership was a poor indicator of water and land usage habits and patterns. We also found that commitment among users in the Middle Rio Grande Valley is to an agricultural lifestyle, not to a community or place. This commitment is conditioned primarily by generational cohort and past experience. If conditions warrant, many would be willing to practice the lifestyle elsewhere. A related finding was that sometimes the pressure to sell was not the putative price of the land, but the taxes on the land. These taxes were, in turn, a function of the level of urbanization of the neighborhood. This urbanization impacted the quality of the agricultural lifestyle. The project also yielded some valuable lessons regarding the model development process. A facilitative and collaborative style (rather than a top-down, directive style) was most productive with the inter-disciplinary , inter-institutional team that worked on the project. This allowed for the emergence of a process model which combined small, discipline- and/or task-specific subgroups with larger, integrating team meetings. The project objective was to develop a model that could be used to run test scenarios in which we explored the potential impact of different policy options. We achieved that objective, although not with the level of success or modeling fidelity which we had hoped for. This report only describes very superficially the results of test scenarios, since more complete analysis of scenarios would require more time and effort. Our greatest obstacle in the successful completion of the project was that required data were sparse, of poor quality, or completely nonexistent. Moreover, we found no similar modeling or research efforts taking place at either the state or local level. This leads to a key finding of this project: that state and local policy decisions regarding land use, development, urbanization, and water resource allocation are being made with minimal data and without the benefit of economic or social policy analysis.

This study investigates the factors that lead countries into conflict. Specifically, political, social and economic factors may offer insight as to how prone a country (or set of countries) may be for inter-country or intra-country conflict. Largely methodological in scope, this study examines the literature for quantitative models that address or attempt to model conflict both in the past, and for future insight. The analysis concentrates specifically on the system dynamics paradigm, not the political science mainstream approaches of econometrics and game theory. The application of this paradigm builds upon the most sophisticated attempt at modeling conflict as a result of system level interactions. This study presents the modeling efforts built on limited data and working literature paradigms, and recommendations for future attempts at modeling conflict.

Abstract not provided.