The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.
Measures of simulation model complexity generally focus on outputs; we propose measuring the complexity of a model’s causal structure to gain insight into its fundamental character. This article introduces tools for measuring causal complexity. First, we introduce a method for developing a model’s causal structure diagram, which characterises the causal interactions present in the code. Causal structure diagrams facilitate comparison of simulation models, including those from different paradigms. Next, we develop metrics for evaluating a model’s causal complexity using its causal structure diagram. We discuss cyclomatic complexity as a measure of the intricacy of causal structure and introduce two new metrics that incorporate the concept of feedback, a fundamental component of causal structure. The first new metric introduced here is feedback density, a measure of the cycle-based interconnectedness of causal structure. The second metric combines cyclomatic complexity and feedback density into a comprehensive causal complexity measure. Finally, we demonstrate these complexity metrics on simulation models from multiple paradigms and discuss potential uses and interpretations. These tools enable direct comparison of models across paradigms and provide a mechanism for measuring and discussing complexity based on a model’s fundamental assumptions and design.
The prevalence of COVID-19 is shaped by behavioral responses to recommendations and warnings. Available information on the disease determines the population’s perception of danger and thus its behavior; this information changes dynamically, and different sources may report conflicting information. We study the feedback between disease, information, and stay-at-home behavior using a hybrid agent-based-system dynamics model that incorporates evolving trust in sources of information. We use this model to investigate how divergent reporting and conflicting information can alter the trajectory of a public health crisis. The model shows that divergent reporting not only alters disease prevalence over time, but also increases polarization of the population’s behaviors and trust in different sources of information.
The causal structure of a simulation is a major determinant of both its character and behavior, yet most methods we use to compare simulations focus only on simulation outputs. We introduce a method that combines graphical representation with information theoretic metrics to quantitatively compare the causal structures of models. The method applies to agent-based simulations as well as system dynamics models and facilitates comparison within and between types. Comparing models based on their causal structures can illuminate differences in assumptions made by the models, allowing modelers to (1) better situate their models in the context of existing work, including highlighting novelty, (2) explicitly compare conceptual theory and assumptions to simulated theory and assumptions, and (3) investigate potential causal drivers of divergent behavior between models. We demonstrate the method by comparing two epidemiology models at different levels of aggregation.
Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world’s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.
This project studied the potential for multiscale group dynamics in complex social systems, including emergent recursive interaction. Current social theory on group formation and interaction focuses on a single scale (individuals forming groups) and is largely qualitative in its explanation of mechanisms. We combined theory, modeling, and data analysis to find evidence that these multiscale phenomena exist, and to investigate their potential consequences and develop predictive capabilities. In this report, we discuss the results of data analysis showing that some group dynamics theory holds at multiple scales. We introduce a new theory on communicative vibration that uses social network dynamics to predict group life cycle events. We discuss a model of behavioral responses to the COVID-19 pandemic that incorporates influence and social pressures. Finally, we discuss a set of modeling techniques that can be used to simulate multiscale group phenomena.
In this paper we consider the effects of corporate hierarchies on innovation spread across multilayer networks, modeled by an elaborated SIR framework. We show that the addition of management layers can significantly improve spreading processes on both random geometric graphs and empirical corporate networks. Additionally, we show that utilizing a more centralized working relationship network rather than a strict administrative network further increases overall innovation reach. In fact, this more centralized structure in conjunction with management layers is essential to both reaching a plurality of nodes and creating a stable adopted community in the long time horizon. Further, we show that the selection of seed nodes affects the final stability of the adopted community, and while the most influential nodes often produce the highest peak adoption, this is not always the case. In some circumstances, seeding nodes near but not in the highest positions in the graph produces larger peak adoption and more stable long-time adoption.
As climate change and human migration accelerate globally, decision-makers are seeking tools that can deepen their understanding of the complex nexus between climate change and human migration. These tools can help to identify populations under pressure to migrate, and to explore proactive policy options and adaptive measures. Given the complexity of factors influencing migration, this article presents a system dynamics-based model that couples migration decision making and behavior with the interacting dynamics of economy, labor, population, violence, governance, water, food, and disease. The regional model is applied here to the test case of migration within and beyond Mali. The study explores potential systems impacts of a range of proactive policy solutions and shows that improving the effectiveness of governance and increasing foreign aid to urban areas have the highest potential of those investigated to reduce the necessity to migrate in the face of climate change.
There is a wealth of psychological theory regarding the drive for individuals to congregate and form social groups, positing that people may organize out of fear, social pressure, or even to manage their self-esteem. We evaluate three such theories for multi-scale validity by studying them not only at the individual scale for which they were originally developed, but also for applicability to group interactions and behavior. We implement this multi-scale analysis using a dataset of communications and group membership derived from a long-running online game, matching the intent behind the theories to quantitative measures that describe players’ behavior. Once we establish that the theories hold for the dataset, we increase the scope to test the theories at the higher scale of group interactions. Despite being formulated to describe individual cognition and motivation, we show that some group dynamics theories hold at the higher level of group cognition and can effectively describe the behavior of joint decision making and higher-level interactions.
This paper seeks to explore the conditions where leaders from open democracies to authoritarian states become more or less popular in response to perceived economic and social threats to society, along with increases in societal (economic and social) hardship and group polarization effects. To further explore these conditions, we used a psycho-social approach to develop a preliminary conceptual model of how the perception of threats, changes in societal conditions, and the polarization of society can concurrently influence the popularity of a government leader.
Improved validation for models of complex systems has been a primary focus over the past year for the Resilience in Complex Systems Research Challenge. This document describes a set of research directions that are the result of distilling those ideas into three categories of research -- epistemic uncertainty, strong tests, and value of information. The content of this document can be used to transmit valuable information to future research activities, update the Resilience in Complex Systems Research Challenge's roadmap, inform the upcoming FY18 Laboratory Directed Research and Development (LDRD) call and research proposals, and facilitate collaborations between Sandia and external organizations. The recommended research directions can provide topics for collaborative research, development of proposals, workshops, and other opportunities.
We created a simulation model to investigate potential links between the actions of violent extremist organizations (VEOs), people in the VEO’s home country, and diaspora communities from that country living in the West. We created this model using the DYMATICA framework, which uses a hybrid cognitive system dynamics modeling strategy to simulate behaviors based on psycho-social theory. Initial results of the model are given, focusing on increases to VEO funding and recruiting resulting from an invasion of the VEO’s home country. Western intervention, prejudice, and economic drivers are also considered.
This report contains the written footprint of a Sandia-hosted workshop held in Albuquerque, New Mexico, June 22-23, 2016 on “Complex Systems Models and Their Applications: Towards a New Science of Verification, Validation and Uncertainty Quantification,” as well as of pre-work that fed into the workshop. The workshop’s intent was to explore and begin articulating research opportunities at the intersection between two important Sandia communities: the complex systems (CS) modeling community, and the verification, validation and uncertainty quantification (VVUQ) community The overarching research opportunity (and challenge) that we ultimately hope to address is: how can we quantify the credibility of knowledge gained from complex systems models, knowledge that is often incomplete and interim, but will nonetheless be used, sometimes in real-time, by decision makers?
The Republic of Estonia faced a series of cyber attacks and riots in 2007 that seemed to be highly coordinated and politically motivated, causing short-lived but substantial impact to Estonia's cyber and economic systems. Short-Term harm from these hybrid incidents led to long-Term improvements and leadership by Estonia in the cyber arena. We created a causal model of these attacks to simulate their dynamics. The model uses the DYMATICA framework, a cognitive-system dynamics structure used to quantify and simulate elicited information from subject matter experts. This historical case study underscores how cyber warfare can be a major threat to modern society, and how it can be combined with information operations and kinetic effects to create further disruption. Given states' potential vulnerability to cyber attacks, a deeper understanding of how to analyze, prevent, defend, and utilize the aftermath of these for improvement to systems is critical, as is insight into the fundamental rationale of the outcomes.
We created a cognition-focused system dynamics model to simulate the dynamics of smoking tendencies based on media influences and communication of opinions. We based this model on the premise that the dynamics of attitudes about smoking can be more deeply understood by combining opinion dynamics with more in-depth psychological models that explicitly explore the root causes of behaviors of interest. Results of the model show the relative effectiveness of two different policies as compared to a baseline: A decrease in advertising spending, and an increase in educational spending. The initial results presented here indicate the utility of this type of simulation for analyzing various policies meant to influence the dynamics of opinions in a population.
Developing nations incur a greater risk to climate change than the developed world due to poorly managed human/natural resources, unreliable infrastructure and brittle governing/economic institutions. These vulnerabilities often give rise to a climate induced “domino effect” of reduced natural resource production-leading to economic hardship, social unrest, and humanitarian crises. Integral to this cascading set of events is increased human migration, leading to the “spillover” of impacts to adjoining areas with even broader impact on global markets and security. Given the complexity of factors influencing human migration and the resultant spill-over effect, quantitative tools are needed to aid policy analysis. Toward this need, a series of migration models were developed along with a system dynamics model of the spillover effect. The migration decision models were structured according to two interacting paths, one that captured long-term “chronic” impacts related to protracted deteriorating quality of life and a second focused on short-term “acute” impacts of disaster and/or conflict. Chronic migration dynamics were modeled for two different cases; one that looked only at emigration but at a national level for the entire world; and a second that looked at both emigration and immigration but focused on a single nation. Model parameterization for each of the migration models was accomplished through regression analysis using decadal data spanning the period 1960-2010. A similar approach was taken with acute migration dynamics except regression analysis utilized annual data sets limited to a shorter time horizon (2001-2013). The system dynamics spillover model was organized around two broad modules, one simulating the decision dynamics of migration and a second module that treats the changing environmental conditions that influence the migration decision. The environmental module informs the migration decision, endogenously simulating interactions/changes in the economy, labor, population, conflict, water, and food. A regional model focused on Mali in western Africa was used as a test case to demonstrate the efficacy of the model.
This project evaluates the effectiveness of moving target defense (MTD) techniques using a new game we have designed, called PLADD, inspired by the game FlipIt [28]. PLADD extends FlipIt by incorporating what we believe are key MTD concepts. We have analyzed PLADD and proven the existence of a defender strategy that pushes a rational attacker out of the game, demonstrated how limited the strategies available to an attacker are in PLADD, and derived analytic expressions for the expected utility of the game’s players in multiple game variants. We have created an algorithm for finding a defender’s optimal PLADD strategy. We show that in the special case of achieving deterrence in PLADD, MTD is not always cost effective and that its optimal deployment may shift abruptly from not using MTD at all to using it as aggressively as possible. We believe our effort provides basic, fundamental insights into the use of MTD, but conclude that a truly practical analysis requires model selection and calibration based on real scenarios and empirical data. We propose several avenues for further inquiry, including (1) agents with adaptive capabilities more reflective of real world adversaries, (2) the presence of multiple, heterogeneous adversaries, (3) computational game theory-based approaches such as coevolution to allow scaling to the real world beyond the limitations of analytical analysis and classical game theory, (4) mapping the game to real-world scenarios, (5) taking player risk into account when designing a strategy (in addition to expected payoff), (6) improving our understanding of the dynamic nature of MTD-inspired games by using a martingale representation, defensive forecasting, and techniques from signal processing, and (7) using adversarial games to develop inherently resilient cyber systems.
Cyber attacks pose a major threat to modern organizations. Little is known about the social aspects of decision making among organizations that face cyber threats, nor do we have empirically-grounded models of the dynamics of cooperative behavior among vulnerable organizations. The effectiveness of cyber defense can likely be enhanced if information and resources are shared among organizations that face similar threats. Three models were created to begin to understand the cognitive and social aspects of cyber cooperation. The first simulated a cooperative cyber security program between two organizations. The second focused on a cyber security training program in which participants interact (and potentially cooperate) to solve problems. The third built upon the first two models and simulates cooperation between organizations in an information-sharing program.
As the US continues its vigilance against distributed, embedded threats, understanding the political and social structure of these groups becomes paramount for predicting and dis- rupting their attacks. Agent-based models (ABMs) serve as a powerful tool to study these groups. While the popularity of social network tools (e.g., Facebook, Twitter) has provided extensive communication data, there is a lack of ne-grained behavioral data with which to inform and validate existing ABMs. Virtual worlds, in particular massively multiplayer online games (MMOG), where large numbers of people interact within a complex environ- ment for long periods of time provide an alternative source of data. These environments provide a rich social environment where players engage in a variety of activities observed between real-world groups: collaborating and/or competing with other groups, conducting battles for scarce resources, and trading in a market economy. Strategies employed by player groups surprisingly re ect those seen in present-day con icts, where players use diplomacy or espionage as their means for accomplishing their goals. In this project, we propose to address the need for ne-grained behavioral data by acquiring and analyzing game data a commercial MMOG, referred to within this report as Game X. The goals of this research were: (1) devising toolsets for analyzing virtual world data to better inform the rules that govern a social ABM and (2) exploring how virtual worlds could serve as a source of data to validate ABMs established for analogous real-world phenomena. During this research, we studied certain patterns of group behavior to compliment social modeling e orts where a signi cant lack of detailed examples of observed phenomena exists. This report outlines our work examining group behaviors that underly what we have termed the Expression-To-Action (E2A) problem: determining the changes in social contact that lead individuals/groups to engage in a particular behavior. Results from our work indicate that virtual worlds have the potential for serving as a proxy in allocating and populating behaviors that would be used within further agent-based modeling studies.
Human and social modeling has emerged as an important research area at Sandia National Laboratories due to its potential to improve national defense-related decision-making in the presence of uncertainty. To learn about which sensitivity analysis techniques are most suitable for models of human behavior, different promising methods were applied to an example model, tested, and compared. The example model simulates cognitive, behavioral, and social processes and interactions, and involves substantial nonlinearity, uncertainty, and variability. Results showed that some sensitivity analysis methods create similar results, and can thus be considered redundant. However, other methods, such as global methods that consider interactions between inputs, can generate insight not gained from traditional methods.