The ground truth program used simulations as test beds for social science research methods. The simulations had known ground truth and were capable of producing large amounts of data. This allowed research teams to run experiments and ask questions of these simulations similar to social scientists studying real-world systems, and enabled robust evaluation of their causal inference, prediction, and prescription capabilities. We tested three hypotheses about research effectiveness using data from the ground truth program, specifically looking at the influence of complexity, causal understanding, and data collection on performance. We found some evidence that system complexity and causal understanding influenced research performance, but no evidence that data availability contributed. The ground truth program may be the first robust coupling of simulation test beds with an experimental framework capable of teasing out factors that determine the success of social science research.
Measures of simulation model complexity generally focus on outputs; we propose measuring the complexity of a model’s causal structure to gain insight into its fundamental character. This article introduces tools for measuring causal complexity. First, we introduce a method for developing a model’s causal structure diagram, which characterises the causal interactions present in the code. Causal structure diagrams facilitate comparison of simulation models, including those from different paradigms. Next, we develop metrics for evaluating a model’s causal complexity using its causal structure diagram. We discuss cyclomatic complexity as a measure of the intricacy of causal structure and introduce two new metrics that incorporate the concept of feedback, a fundamental component of causal structure. The first new metric introduced here is feedback density, a measure of the cycle-based interconnectedness of causal structure. The second metric combines cyclomatic complexity and feedback density into a comprehensive causal complexity measure. Finally, we demonstrate these complexity metrics on simulation models from multiple paradigms and discuss potential uses and interpretations. These tools enable direct comparison of models across paradigms and provide a mechanism for measuring and discussing complexity based on a model’s fundamental assumptions and design.
The causal structure of a simulation is a major determinant of both its character and behavior, yet most methods we use to compare simulations focus only on simulation outputs. We introduce a method that combines graphical representation with information theoretic metrics to quantitatively compare the causal structures of models. The method applies to agent-based simulations as well as system dynamics models and facilitates comparison within and between types. Comparing models based on their causal structures can illuminate differences in assumptions made by the models, allowing modelers to (1) better situate their models in the context of existing work, including highlighting novelty, (2) explicitly compare conceptual theory and assumptions to simulated theory and assumptions, and (3) investigate potential causal drivers of divergent behavior between models. We demonstrate the method by comparing two epidemiology models at different levels of aggregation.
Social systems are uniquely complex and difficult to study, but understanding them is vital to solving the world’s problems. The Ground Truth program developed a new way of testing the research methods that attempt to understand and leverage the Human Domain and its associated complexities. The program developed simulations of social systems as virtual world test beds. Not only were these simulations able to produce data on future states of the system under various circumstances and scenarios, but their causal ground truth was also explicitly known. Research teams studied these virtual worlds, facilitating deep validation of causal inference, prediction, and prescription methods. The Ground Truth program model provides a way to test and validate research methods to an extent previously impossible, and to study the intricacies and interactions of different components of research.
This project studied the potential for multiscale group dynamics in complex social systems, including emergent recursive interaction. Current social theory on group formation and interaction focuses on a single scale (individuals forming groups) and is largely qualitative in its explanation of mechanisms. We combined theory, modeling, and data analysis to find evidence that these multiscale phenomena exist, and to investigate their potential consequences and develop predictive capabilities. In this report, we discuss the results of data analysis showing that some group dynamics theory holds at multiple scales. We introduce a new theory on communicative vibration that uses social network dynamics to predict group life cycle events. We discuss a model of behavioral responses to the COVID-19 pandemic that incorporates influence and social pressures. Finally, we discuss a set of modeling techniques that can be used to simulate multiscale group phenomena.
It is widely believed that one's peers influence product adoption behaviors. This relationship has been linked to the number of signals a decision-maker receives in a social network. But it is unclear if these same principles hold when the "pattern" by which it receives these signals vary and when peer influence is directed towards choices which are not optimal. To investigate that, we manipulate social signal exposure in an online controlled experiment using a game with human participants. Each participant in the game decides among choices with differing utilities. We observe the following: (1) even in the presence of monetary risks and previously acquired knowledge of the choices, decision-makers tend to deviate from the obvious optimal decision when their peers make a similar decision which we call the influence decision, (2) when the quantity of social signals vary over time, the forwarding probability of the influence decision and therefore being responsive to social influence does not necessarily correlate proportionally to the absolute quantity of signals. To better understand how these rules of peer influence could be used in modeling applications of real world diffusion and in networked environments, we use our behavioral findings to simulate spreading dynamics in real world case studies. We specifically try to see how cumulative influence plays out in the presence of user uncertainty and measure its outcome on rumor diffusion, which we model as an example of sub-optimal choice diffusion. Together, our simulation results indicate that sequential peer effects from the influence decision overcomes individual uncertainty to guide faster rumor diffusion over time. However, when the rate of diffusion is slow in the beginning, user uncertainty can have a substantial role compared to peer influence in deciding the adoption trajectory of a piece of questionable information.
SIGNAL is a first of its kind experimental wargame developed as part of the Project on Nuclear Gaming (PoNG). In this document we describe the rules and game mechanics associated with the online version of SIGNAL created by team members from the University of California, Berkeley, Sandia National Laboratories, and Lawrence Livermore National Laboratory and sponsored by the Carnegie Corporation of New York. The game was developed as part of a larger research project to develop the experimental wargaming methodology and explore its use on a model scenario: the impact of various military capabilities on conflict escalation dynamics. We discuss the results of this research in a forthcoming paper that will include this manual as an appendix. It is our hope that this manual will both contribute to our players' understanding of the game prior to play and that it will allow for replication of the SIGNAL game environment for future research purposes. The manual begins by introducing the terminology used throughout the document. It then outlines the technical requirements required to run SIGNAL. The following section provides a description of the map, resources, infrastructure, tokens, and action cards used in the game environment. The manual then describes the user interface including the chat functions, trade mechanism, currency and population counts necessary for players to plan their actions. It then turns to the sequence of player actions in the game describing the signaling, action, and upkeep phases that comprise each round of play. It then outlines the use of diplomacy including alliances with minor states and trade between players. The manual also describes the process for scoring the game and determining the winner. The manual concludes with tips for players to remember as they embark upon playing the game.
We develop a methodology for comparing two or more agent-based models that are developed for the same domain, but may differ in the particular data sets (e.g., geographical regions) to which they are applied, and in the structure of the model. Our approach is to learn a response surface in the common parameter space of the models and compare the regions corresponding to qualitatively different behaviors in the models. As an example, we develop an active learning algorithm to learn phase transition boundaries in contagion processes in order to compare two agent-based models of rooftop solar panel adoption.
There is a wealth of psychological theory regarding the drive for individuals to congregate and form social groups, positing that people may organize out of fear, social pressure, or even to manage their self-esteem. We evaluate three such theories for multi-scale validity by studying them not only at the individual scale for which they were originally developed, but also for applicability to group interactions and behavior. We implement this multi-scale analysis using a dataset of communications and group membership derived from a long-running online game, matching the intent behind the theories to quantitative measures that describe players’ behavior. Once we establish that the theories hold for the dataset, we increase the scope to test the theories at the higher scale of group interactions. Despite being formulated to describe individual cognition and motivation, we show that some group dynamics theories hold at the higher level of group cognition and can effectively describe the behavior of joint decision making and higher-level interactions.
National security decisions are driven by complex, interconnected contextual, individual, and strategic variables. Modeling and simulation tools are often used to identify relevant patterns, which can then be shaped through policy remedies. In the paper to follow, however, we argue that models of these scenarios may be prone to the complexity-scarcity gap, in which relevant scenarios are too complex to model from first principles and data from historical scenarios are too sparse-making it difficult to draw representative conclusions. The result are models that are either too simple or are unduly biased by the assumptions of the analyst. We outline a new method of quantitative inquiry-experimental wargaming-as a means to bridge the complexity-scarcity gap that offers human-generated, empirical data to inform a variety of model and simulation tasks (model building, calibration, testing, and validation). Below, we briefly describe SIGNAL-our first-of-a-kind experimental wargame designed to study strategic stability in conflict settings with nuclear weapons. We then highlight the potential utility of this data for modeling and simulation efforts in the future using this data.
It is widely believed that the adoption behavior of a decision-maker in a social network is related to the number of signals it receives from its peers in the social network. It is unclear if these same principles hold when the “pattern” by which they receive these signals vary and when potential decisions have different utilities. To investigate that, we manipulate social signal exposure in an online controlled experiment with human participants. Specifically, we change the number of signals and the pattern through which participants receive them over time. We analyze its effect through a controlled game where each participant makes a decision to select one option when presented with six choices with differing utilities, with one choice having the most utility. We avoided network effects by holding the neighborhood network of the users constant. Over multiple rounds of the game, we observe the following: (1) even in the presence of monetary risks and previously acquired knowledge of the six choices, decision-makers tend to deviate from the obvious optimal decision when their peers make similar choices, (2) when the quantity of social signals vary over time, the probability that a participant selects the decision similar to the one reflected by the social signals and therefore being responsive to social influence does not necessarily correlate proportionally to the absolute quantity of signals and (3) an early subjugation to higher quantity of peer social signals turned out to be a more effective strategy of social influence when aggregated over the rounds.
While social media enables users and organizations to obtain useful information about technology like software and security feature usage, it can also allow an adversary to exploit users by obtaining information from them or influencing them towards injurious decisions. Prior research indicates that security technology choices are subject to social influence and that these decisions are often influenced by the peer decisions and number of peers in a user’s network. In this study we investigated whether peer influence dictates users’ decisions by manipulating social signals from peers in an online, controlled experiment. Human participants recruited from Amazon Mechanical Turk played a multi-round game in which they selected a security technology from among six of differing utilities. We observe that at the end of the game, a strategy to expose users to high quantity of peer signals reflecting suboptimal choices, in the later stages of the game successfully influences users to deviate from the optimal security technology. This strategy influences almost 1.5 times the number of users with respect to the strategy where users receive constant low quantity of similar peer signals in all rounds of the game.
Much has been written on the potential for games to enhance our ability to study complex systems. In this chapter we focus on how we can use games to study national security issues. We reflect on the benefits of using games and the inherent difficulties that we must address. As a means of grounding the discussion, we will present a case study of a retrospective analysis of gaming data.
In order to understand the effect of economic interdependence on conflict and on deterrents to conflict, and to assess the viability of online games as experiments to perform research, an online serious game was used to gather data on economic, political, and military factors in the game setting. These data were operationalized in forms analogous to variables from the real-world Militarized Interstate Disputes (MIDs) dataset. A set of economic predictor variables was analyzed using linear mixed effects regression models in an attempt to discover relationships between the predictor variables and conflict outcomes. Differences between the online game results and results from the real world are discussed.
The doctrine of nuclear deterrence and a belief in its importance underpins many aspects of United States policy; it informs strategic force structures within the military, incentivizes multi-billion-dollar weapon-modernization programs within the Department of Energy, and impacts international alliances with the 29 member states of the North Atlantic Treaty Organization (NATO). The doctrine originally evolved under the stewardship of some of the most impressive minds of the twentieth century, including the physicist and H-bomb designer Herman Kahn, the Nobel Prize-winning economist Thomas Schelling, and the preeminent political scientist and diplomat Henry Kissinger.
The wealth of information available on seemingly every topic creates a considerable challenge both for information providers trying to rise above the noise and discerning individuals trying to find relevant, trustworthy information. We approach this information problem by investigating how passive versus interactive information interventions can impact the antecedents of behavior change using the context of solar energy adoption, where persistent information gaps are known to reduce market potential. We use two experiments to investigate the impact of both passive and interactive approaches to information delivery on the antecedents (attitudes, subjective norms, and perceived behavioral control in the Theory of Planned Behavior) of intentions and behavior, as well as their effect on intentions and behavior directly. The passive information randomized control trial delivered via Amazon Mechanical Turk tests the effectiveness of delivering the same content in a single message versus multiple shorter messages. The interactive information delivery uses an online (mobile and PC) trivia-style gamification platform. Both experiments use the same content and are carried out over a two-week time period. Our findings suggest that interactive, gamified information has greater impact than passive information, and that shorter multiple messages of passive information are more effective than a single passive message.
People use social media resources like Twitter, Facebook, forums etc. to shareand discuss various activities or topics. By aggregating topic trends acrossmany individuals using these services, we seek to construct a richer profileof a person's activities and interests as well as provide a broader context ofthose activities. This profile may then be used in a variety of ways tounderstand groups as a collection of interests and affinities and anindividual's participation in those groups. Our approach considers that muchof these data will be unstructured, free-form text. By analyzing free-form text directly, we may be able to gain an implicit grouping ofindividuals with shared interests based on shared conversation, and not onexplicit social software linking them. In this paper, we discuss aproof-of-concept application called Grandmaster built to pull short sections oftext, a person's comments or Twitter posts, together by analysis andvisualization to allow a gestalt understanding of the full collection of allindividuals: how groups are similar and how they differ, based on theirtext inputs.
Alisa M. Rogers University of Georgia The US faces persistent, distributed threats from malevolent individuals, groups and or- ganizations around the world. Computational Social Models (CSMs) help anticipate the dynamics and behaviors of these actors by modeling the behavior and interactions of indi- viduals, groups and organizations. For strategic planners to trust the results of CSMs, they must have confidence in the validity of the models. Establishing validity before model use will enhance confidence and reduce the risk of error. One problem with validation is design- ing an appropriate controlled test of the model, similar to the testing of physical models. Lab experiments can do this, but are often limited to small numbers of subjects, with low subject diversity and are often in a contrived environment. Natural studies attempt to test models by gathering large-scale observational data (e.g., social media) however this loses the controlled aspect. We propose a new approach to run large-scale, controlled online ex- periments on diverse populations. Using Amazon Mechanical Turk, a crowdsourcing tool, we will draw large populations into controlled experiments in a manner that was not possible just a few years ago. In this report we describe the "Controlled, Large Online Social Experimentation (CLOSE)" platform -- a prototype platform develop to conduct online social experiments. Through an extensive survey we find that online subject pools can be recruited to participate in longitudinal online social experiments. We describe the characteristics of these subject pools and their suitability for longitudinal online experiments.
People use social media resources like Twitter, Facebook, forums etc. to share and discuss various activities or topics. By aggregating topic trends across many individuals using these services, we seek to construct a richer profile of a person’s activities and interests as well as provide a broader context of those activities. This profile may then be used in a variety of ways to understand groups as a collection of interests and affinities and an individual’s participation in those groups. Our approach considers that much of these data will be unstructured, free-form text. By analyzing free-form text directly, we may be able to gain an implicit grouping of individuals with shared interests based on shared conversation, and not on explicit social software linking them. In this paper, we discuss a proof-of-concept application called Grandmaster built to pull short sections of text, a person’s comments or Twitter posts, together by analysis and visualization to allow a gestalt understanding of the full collection of all individuals: how groups are similar and how they differ, based on their text inputs.
Agent-based modeling is commonly used for studying complex system properties emergent from interactions among many agents. We present a novel data-driven agent-based modeling framework applied to forecasting individual and aggregate residential rooftop solar adoption in San Diego county. Our first step is to learn a model of individual agent behavior from combined data of individual adoption characteristics and property assessment. We then construct an agent-based simulation with the learned model embedded in artificial agents, and proceed to validate it using a holdout sequence of collective adoption decisions. We demonstrate that the resulting agent-based model successfully forecasts solar adoption trends and provides a meaningful quantification of uncertainty about its predictions. We utilize our model to optimize two classes of policies aimed at spurring solar adoption: one that subsidizes the cost of adoption, and another that gives away free systems to low-income house-holds. We find that the optimal policies derived for the latter class are significantly more efficacious, whereas the policies similar to the current California Solar Initiative incentive scheme appear to have a limited impact on overall adoption trends.
Massively-multiplayer online games (MMOGs) can serve asa unique laboratory for studying large-scale human behaviors. However,one question that often arises is whether the observed behavior is specificto the game world and its winning conditions. This paper studiesthe nature of conflict and communication across two game worlds thathave different game objectives. We compare and contrast the structureof attack networks with trade and communication networks. Similar toreal-life, social structures play a significant role in the likelihood of interplayerconflict.
Attitude diffusion is when "attitudes" (general, relatively enduring evaluative responses to a topic) spread through a population. Attitudes play an incredibly important role in human decision making and are a critical part of social psychology. However, existing models of diffusion do not account for key differentiating aspects of attitudes. We develop the "Multi-Agent, Multi-Attitude" (MAMA) model which incorporates several of these key factors: (1) multiple, interacting attitudes; (2) social influence between individuals; and (3) media influence. All three components have strong support from the social science community. Using the MAMA model, we study influence maximization in a attitude diffusion setting where media influence is possible - we show that strategic manipulation of the media can lead to statistically significant decreases in diffusion of attitudes.
Amazon Mechanical Turk (AMT) has become a powerful tool for social scientists due to its inexpensiveness, ease of use, and ability to attract large numbers of workers. While the subject pool is diverse, there are numerous questions regarding the composition of the workers as a function of when the “Human Intelligence Task”(HIT) is posted. Given the “queue” nature of HITs and the disparity in geography of participants, it is natural to wonder whether HIT posting time/day can have an impact on the population that is sampled.We address this question using a panel survey on AMT and show (surprisingly) that except for gender, there is no statistically significant difference in terms of demographics characteristics as a function of HIT posting time.
As the US continues its vigilance against distributed, embedded threats, understanding the political and social structure of these groups becomes paramount for predicting and dis- rupting their attacks. Agent-based models (ABMs) serve as a powerful tool to study these groups. While the popularity of social network tools (e.g., Facebook, Twitter) has provided extensive communication data, there is a lack of ne-grained behavioral data with which to inform and validate existing ABMs. Virtual worlds, in particular massively multiplayer online games (MMOG), where large numbers of people interact within a complex environ- ment for long periods of time provide an alternative source of data. These environments provide a rich social environment where players engage in a variety of activities observed between real-world groups: collaborating and/or competing with other groups, conducting battles for scarce resources, and trading in a market economy. Strategies employed by player groups surprisingly re ect those seen in present-day con icts, where players use diplomacy or espionage as their means for accomplishing their goals. In this project, we propose to address the need for ne-grained behavioral data by acquiring and analyzing game data a commercial MMOG, referred to within this report as Game X. The goals of this research were: (1) devising toolsets for analyzing virtual world data to better inform the rules that govern a social ABM and (2) exploring how virtual worlds could serve as a source of data to validate ABMs established for analogous real-world phenomena. During this research, we studied certain patterns of group behavior to compliment social modeling e orts where a signi cant lack of detailed examples of observed phenomena exists. This report outlines our work examining group behaviors that underly what we have termed the Expression-To-Action (E2A) problem: determining the changes in social contact that lead individuals/groups to engage in a particular behavior. Results from our work indicate that virtual worlds have the potential for serving as a proxy in allocating and populating behaviors that would be used within further agent-based modeling studies.
We consider the question of predicting solar adoption using demographic, economic, peer effect and predicted system characteristic features. We use data from San Diego county to evaluate both discrete and continuous models. Additionally, we consider three types of sensitivity analysis to identify which features seem to have the greatest effect on prediction accuracy.
Attitudes play a significant role in determining how individuals process information and behave. In this paper we have developed a new computational model of population wide attitude change that captures the social level: how individuals interact and communicate information, and the cognitive level: how attitudes and concept interact with each other. The model captures the cognitive aspect by representing each individuals as a parallel constraint satisfaction network. The dynamics of this model are explored through a simple attitude change experiment where we vary the social network and distribution of attitudes in a population.