Publications

{Developing a System for Testing Computational Social Models using Amazon Mechanical Turk

Lakkaraju, Kiran L.; Rogers, Alisa R.

Alisa M. Rogers University of Georgia The US faces persistent, distributed threats from malevolent individuals, groups and or- ganizations around the world. Computational Social Models (CSMs) help anticipate the dynamics and behaviors of these actors by modeling the behavior and interactions of indi- viduals, groups and organizations. For strategic planners to trust the results of CSMs, they must have confidence in the validity of the models. Establishing validity before model use will enhance confidence and reduce the risk of error. One problem with validation is design- ing an appropriate controlled test of the model, similar to the testing of physical models. Lab experiments can do this, but are often limited to small numbers of subjects, with low subject diversity and are often in a contrived environment. Natural studies attempt to test models by gathering large-scale observational data (e.g., social media) however this loses the controlled aspect. We propose a new approach to run large-scale, controlled online ex- periments on diverse populations. Using Amazon Mechanical Turk, a crowdsourcing tool, we will draw large populations into controlled experiments in a manner that was not possible just a few years ago. In this report we describe the "Controlled, Large Online Social Experimentation (CLOSE)" platform -- a prototype platform develop to conduct online social experiments. Through an extensive survey we find that online subject pools can be recruited to participate in longitudinal online social experiments. We describe the characteristics of these subject pools and their suitability for longitudinal online experiments.