Publications

The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.

Abstract not provided.

Abstract not provided.

Measures of simulation model complexity generally focus on outputs; we propose measuring the complexity of a model’s causal structure to gain insight into its fundamental character. This article introduces tools for measuring causal complexity. First, we introduce a method for developing a model’s causal structure diagram, which characterises the causal interactions present in the code. Causal structure diagrams facilitate comparison of simulation models, including those from different paradigms. Next, we develop metrics for evaluating a model’s causal complexity using its causal structure diagram. We discuss cyclomatic complexity as a measure of the intricacy of causal structure and introduce two new metrics that incorporate the concept of feedback, a fundamental component of causal structure. The first new metric introduced here is feedback density, a measure of the cycle-based interconnectedness of causal structure. The second metric combines cyclomatic complexity and feedback density into a comprehensive causal complexity measure. Finally, we demonstrate these complexity metrics on simulation models from multiple paradigms and discuss potential uses and interpretations. These tools enable direct comparison of models across paradigms and provide a mechanism for measuring and discussing complexity based on a model’s fundamental assumptions and design.

Abstract not provided.

The causal structure of a simulation is a major determinant of both its character and behavior, yet most methods we use to compare simulations focus only on simulation outputs. We introduce a method that combines graphical representation with information theoretic metrics to quantitatively compare the causal structures of models. The method applies to agent-based simulations as well as system dynamics models and facilitates comparison within and between types. Comparing models based on their causal structures can illuminate differences in assumptions made by the models, allowing modelers to (1) better situate their models in the context of existing work, including highlighting novelty, (2) explicitly compare conceptual theory and assumptions to simulated theory and assumptions, and (3) investigate potential causal drivers of divergent behavior between models. We demonstrate the method by comparing two epidemiology models at different levels of aggregation.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world’s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.

Wargames are a common tool for investigating complex conflict scenarios and have a long history of informing military and strategic study. Historically, these games have often been one offs, may not rigorously collect data, and have been built primarily for exploration rather than developing data-driven analytical conclusions. Experimental wargaming, a new wargaming approach that employs the basic principles of experimental design to facilitate an objective basis for exploring fundamental research questions around human behavior (such as understanding conflict escalation), is a potential tool that can be used in combination with existing wargaming approaches. The Project on Nuclear Gaming, a consortium involving the University of California, Berkeley, Sandia National Laboratories, and Lawrence Livermore National Laboratory, developed an experimental wargame, SIGNAL, to explore questions surrounding conflict escalation and strategic stabil-ity in the nuclear context. To date, the SIGNAL experimental wargame has been played hundreds of times by thousands of players from around the world, creating the largest data-base of wargame data for academic purposes known to the authors. This paper discusses the design of SIGNAL, focusing on how the principles of experimental design influenced this design.

This project studied the potential for multiscale group dynamics in complex social systems, including emergent recursive interaction. Current social theory on group formation and interaction focuses on a single scale (individuals forming groups) and is largely qualitative in its explanation of mechanisms. We combined theory, modeling, and data analysis to find evidence that these multiscale phenomena exist, and to investigate their potential consequences and develop predictive capabilities. In this report, we discuss the results of data analysis showing that some group dynamics theory holds at multiple scales. We introduce a new theory on communicative vibration that uses social network dynamics to predict group life cycle events. We discuss a model of behavioral responses to the COVID-19 pandemic that incorporates influence and social pressures. Finally, we discuss a set of modeling techniques that can be used to simulate multiscale group phenomena.

We describe the opportunities and challenges we faced when developing SIGNAL, an experimental wargame that was deployed as a distributed wargame.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

SIGNAL is a first of its kind experimental wargame developed as part of the Project on Nuclear Gaming (PoNG). In this document we describe the rules and game mechanics associated with the online version of SIGNAL created by team members from the University of California, Berkeley, Sandia National Laboratories, and Lawrence Livermore National Laboratory and sponsored by the Carnegie Corporation of New York. The game was developed as part of a larger research project to develop the experimental wargaming methodology and explore its use on a model scenario: the impact of various military capabilities on conflict escalation dynamics. We discuss the results of this research in a forthcoming paper that will include this manual as an appendix. It is our hope that this manual will both contribute to our players' understanding of the game prior to play and that it will allow for replication of the SIGNAL game environment for future research purposes. The manual begins by introducing the terminology used throughout the document. It then outlines the technical requirements required to run SIGNAL. The following section provides a description of the map, resources, infrastructure, tokens, and action cards used in the game environment. The manual then describes the user interface including the chat functions, trade mechanism, currency and population counts necessary for players to plan their actions. It then turns to the sequence of player actions in the game describing the signaling, action, and upkeep phases that comprise each round of play. It then outlines the use of diplomacy including alliances with minor states and trade between players. The manual also describes the process for scoring the game and determining the winner. The manual concludes with tips for players to remember as they embark upon playing the game.

It is widely believed that one's peers influence product adoption behaviors. This relationship has been linked to the number of signals a decision-maker receives in a social network. But it is unclear if these same principles hold when the "pattern" by which it receives these signals vary and when peer influence is directed towards choices which are not optimal. To investigate that, we manipulate social signal exposure in an online controlled experiment using a game with human participants. Each participant in the game decides among choices with differing utilities. We observe the following: (1) even in the presence of monetary risks and previously acquired knowledge of the choices, decision-makers tend to deviate from the obvious optimal decision when their peers make a similar decision which we call the influence decision, (2) when the quantity of social signals vary over time, the forwarding probability of the influence decision and therefore being responsive to social influence does not necessarily correlate proportionally to the absolute quantity of signals. To better understand how these rules of peer influence could be used in modeling applications of real world diffusion and in networked environments, we use our behavioral findings to simulate spreading dynamics in real world case studies. We specifically try to see how cumulative influence plays out in the presence of user uncertainty and measure its outcome on rumor diffusion, which we model as an example of sub-optimal choice diffusion. Together, our simulation results indicate that sequential peer effects from the influence decision overcomes individual uncertainty to guide faster rumor diffusion over time. However, when the rate of diffusion is slow in the beginning, user uncertainty can have a substantial role compared to peer influence in deciding the adoption trajectory of a piece of questionable information.

Abstract not provided.

Abstract not provided.

There is a wealth of psychological theory regarding the drive for individuals to congregate and form social groups, positing that people may organize out of fear, social pressure, or even to manage their self-esteem. We evaluate three such theories for multi-scale validity by studying them not only at the individual scale for which they were originally developed, but also for applicability to group interactions and behavior. We implement this multi-scale analysis using a dataset of communications and group membership derived from a long-running online game, matching the intent behind the theories to quantitative measures that describe players’ behavior. Once we establish that the theories hold for the dataset, we increase the scope to test the theories at the higher scale of group interactions. Despite being formulated to describe individual cognition and motivation, we show that some group dynamics theories hold at the higher level of group cognition and can effectively describe the behavior of joint decision making and higher-level interactions.

We develop a methodology for comparing two or more agent-based models that are developed for the same domain, but may differ in the particular data sets (e.g., geographical regions) to which they are applied, and in the structure of the model. Our approach is to learn a response surface in the common parameter space of the models and compare the regions corresponding to qualitatively different behaviors in the models. As an example, we develop an active learning algorithm to learn phase transition boundaries in contagion processes in order to compare two agent-based models of rooftop solar panel adoption.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

We report that over the past century, and particularly since the outset of the Cold War, wargames (interactive simulations used to evaluate aspects of tactics, operations, and strategy) have become an integral means for militaries and policy-makers to evaluate how strategic decisions are made related to nuclear weapons strategy and international security. Furthermore, these methods have also been applied beyond the military realm, to examine phenomena as varied as elections, government policy, international trade, and supply-chain mechanics. Today, a renewed focus on wargaming combined with access to sophisticated and inexpensive drag-and-drop digital game development frameworks and new cloud computing architectures have democratized the ability to enable massive multiplayer gaming experiences. With the integration of simulation tools and experimental methods from a variety of social science disciplines, a science-based experimental gaming approach has the potential to transform the insights generated from gaming by creating human-derived, large-n datasets for replicable, quantitative analysis. In the following, we outline challenges associated with contemporary simulation and wargaming tools, investigate where scholars have searched for game data, and explore the utility of new experimental gaming and data analysis methods in both policy-making and academic settings.

Abstract not provided.

In order to understand the effect of economic interdependence on conflict and on deterrents to conflict, and to assess the viability of online games as experiments to perform research, an online serious game was used to gather data on economic, political, and military factors in the game setting. These data were operationalized in forms analogous to variables from the real-world Militarized Interstate Disputes (MIDs) dataset. A set of economic predictor variables was analyzed using linear mixed effects regression models in an attempt to discover relationships between the predictor variables and conflict outcomes. Differences between the online game results and results from the real world are discussed.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

Abstract not provided.

As concerns with cyber security and network protection increase, there is a greater need for organizations to deploy state-of-the-art technology to keep their cyber information safe. However, foolproof cyber security and network protection are a difficult feat since a security breach can be caused simply by a single employee who unknowingly succumbs to a cyber threat. It is critical for an organization’s workforce to holistically adopt cyber technologies that enable enhanced protection, help ward off cyber threats, and are efficient at encouraging human behavior towards safer cyber practices. It is also crucial for the workforce, once they have adopted cyber technologies, to remain consistent and thoughtful in their use of these technologies to keep resistance strong against cyber threats and vulnerabilities. Adoption of cyber technology can be difficult. Many organizations struggle with their workforce adopting newly-introduced cyber technologies, even when the technologies themselves have proven to be worthy solutions. Research, especially in the domain of cognitive science and the human dimension, has sought to understand how technology adoption works and can be leveraged. This paper reviews what empirical literature has found regarding cyber technology adoption, the current research gaps, and how non-research based efforts can influence adoption. Focusing on current efforts accomplished by a government-sponsored activity entitled “ACT” (Adoption of Cybersecurity Technologies), the aim of this paper is to empirically study cyber technology adoption to better understand how to influence operational adoption across the government-sector as well as how what can be done to develop a model that enables cyber technology adoption.

The wealth of information available on seemingly every topic creates a considerable challenge both for information providers trying to rise above the noise and discerning individuals trying to find relevant, trustworthy information. We approach this information problem by investigating how passive versus interactive information interventions can impact the antecedents of behavior change using the context of solar energy adoption, where persistent information gaps are known to reduce market potential. We use two experiments to investigate the impact of both passive and interactive approaches to information delivery on the antecedents (attitudes, subjective norms, and perceived behavioral control in the Theory of Planned Behavior) of intentions and behavior, as well as their effect on intentions and behavior directly. The passive information randomized control trial delivered via Amazon Mechanical Turk tests the effectiveness of delivering the same content in a single message versus multiple shorter messages. The interactive information delivery uses an online (mobile and PC) trivia-style gamification platform. Both experiments use the same content and are carried out over a two-week time period. Lastly, our findings suggest that interactive, gamified information has greater impact than passive information, and that shorter multiple messages of passive information are more effective than a single passive message.

Agent-based modeling is commonly used for studying complex system properties emergent from interactions among agents. However, agent-based models are often not developed explicitly for prediction, and are generally not validated as such. We therefore present a novel data-driven agent-based modeling framework, in which individual behavior model is learned by machine learning techniques, deployed in multi-agent systems and validated using a holdout sequence of collective adoption decisions. We apply the framework to forecasting individual and aggregate residential rooftop solar adoption in San Diego county and demonstrate that the resulting agent-based model successfully forecasts solar adoption trends and provides a meaningful quantification of uncertainty about its predictions. Meanwhile, we construct a second agent-based model, with its parameters calibrated based on mean square error of its fitted aggregate adoption to the ground truth. Our result suggests that our data-driven agent-based approach based on maximum likelihood estimation substantially outperforms the calibrated agent-based model. Seeing advantage over the state-of-the-art modeling methodology, we utilize our agent-based model to aid search for potentially better incentive structures aimed at spurring more solar adoption. Although the impact of solar subsidies is rather limited in our case, our study still reveals that a simple heuristic search algorithm can lead to more effective incentive plans than the current solar subsidies in San Diego County and a previously explored structure. Finally, we examine an exclusive class of policies that gives away free systems to low-income households, which are shown significantly more efficacious than any incentive-based policies we have analyzed to date.

Abstract not provided.

People use social media resources like Twitter, Facebook, forums etc. to share and discuss various activities or topics. By aggregating topic trends across many individuals using these services, we seek to construct a richer profile of a person’s activities and interests as well as provide a broader context of those activities. This profile may then be used in a variety of ways to understand groups as a collection of interests and affinities and an individual’s participation in those groups. Our approach considers that much of these data will be unstructured, free-form text. By analyzing free-form text directly, we may be able to gain an implicit grouping of individuals with shared interests based on shared conversation, and not on explicit social software linking them. In this paper, we discuss a proof-of-concept application called Grandmaster built to pull short sections of text, a person’s comments or Twitter posts, together by analysis and visualization to allow a gestalt understanding of the full collection of all individuals: how groups are similar and how they differ, based on their text inputs.

People use social media resources like Twitter, Facebook, forums etc. to share and discuss various activities or topics. By aggregating topic trends across many individuals using these services, we seek to construct a richer profile of a person’s activities and interests as well as provide a broader context of those activities. This profile may then be used in a variety of ways to understand groups as a collection of interests and affinities and an individual’s participation in those groups. Our approach considers that much of these data will be unstructured, free-form text. By analyzing free-form text directly, we may be able to gain an implicit grouping of individuals with shared interests based on shared conversation, and not on explicit social software linking them. In this paper, we discuss a proof-of-concept application called Grandmaster built to pull short sections of text, a person’s comments or Twitter posts, together by analysis and visualization to allow a gestalt understanding of the full collection of all individuals: how groups are similar and how they differ, based on their text inputs.