Publications

Comparing reproduced cyber experimentation studies across different emulation testbeds

Tarman, Thomas D.; Rollins, Trevor; Swiler, Laura P.; Cruz, Gerardo C.; Vugrin, Eric D.; Huang, Hao; Sahu, Abhijeet; Wlazlo, Patrick; Goulart, Ana; Davis, Kate

Cyber testbeds provide an important mechanism for experimentally evaluating cyber security performance. However, as an experimental discipline, reproducible cyber experimentation is essential to assure valid, unbiased results. Even minor differences in setup, configuration, and testbed components can have an impact on the experiments, and thus, reproducibility of results. This paper documents a case study in reproducing an earlier emulation study, with the reproduced emulation experiment conducted by a different research group on a different testbed. We describe lessons learned as a result of this process, both in terms of the reproducibility of the original study and in terms of the different testbed technologies used by both groups. This paper also addresses the question of how to compare results between two groups' experiments, identifying candidate metrics for comparison and quantifying the results in this reproduction study.