Many teams struggle to adapt and right-size software engineering best practices for quality assurance to fit their context. Introducing software quality is not usually framed in a way that motivates teams to take action, thus resulting in it becoming a “check the box for compliance” activity instead of a cultural practice that values software quality and the effort to achieve it. When and how can we provide effective incentives for software teams to adopt and integrate meaningful and enduring software quality practices? Here, we explored this question through a persona-based ideation exercise at the 2021 Collegeville Workshop on Scientific Software in which we created three unique personas that represent different scientific software developer perspectives.
Although many software teams across the laboratories comply with yearly software quality engineering (SQE) assessments, the practice of introducing quality into each phase of the software lifecycle, or the team processes, may vary substantially. Even with the support of a quality engineer, many teams struggle to adapt and right-size software engineering best practices in quality to fit their context, and these activities aren’t framed in a way that motivates teams to take action. In short, software quality is often a “check the box for compliance” activity instead of a cultural practice that both values software quality and knows how to achieve it. In this report, we present the results of our 6600 VISTA Innovation Tournament project, "Incentivizing and Motivating High Confidence and Research Software Teams to Adopt the Practice of Quality." We present our findings and roadmap for future work based on 1) a rapid review of relevant literature, 2) lessons learned from an internal design thinking workshop, and 3) an external Collegeville 2021 workshop. These activities provided an opportunity for team ideation and community engagement/feedback. Based on our findings, we believe a coordinated effort (e.g. strategic communication campaign) aimed at diffusing the innovation of the practice of quality across Sandia National Laboratories could over time effect meaningful organizational change. As such, our roadmap addresses strategies for motivating and incentivizing individuals ranging from early career to seasoned software developers/scientists.
In response to the COVID-19 pandemic, the Exascale Computing Project’s (ECP) Interoperable Design of Extreme-scale Application Software (IDEAS) productivity team launched the panel series Strategies for Working Remotely to facilitate informal, cross-organizational dialog in the absence of face-to-face meetings. In a time of pandemic, organizations increasingly need to reach across perceived boundaries to learn from each other, so that we can move beyond stand-alone silos to more connected multidisciplinary and multiorganizational configurations. The present paper argues that the unplanned transition to remote work, overuse of electronic communication, and need to unlearn habits associated with an overreliance on face-to-face, created unique opportunities to learn from the situation and accelerate cross-institutional cooperation and collaboration through online community dialog facilitated by informal panel discussions. Recommendations for facilitating online panel discussions to foster cross-organizational dialog are provided by applying the Simulation Experience Design Method.
Productivity and Sustainability Improvement Planning (PSIP) is a lightweight, iterative workflow that allows software development teams to identify development bottlenecks and track progress to overcome them. In this paper, we present an overview of PSIP and how it compares to other software process improvement (SPI) methodologies, and provide two case studies that describe how the use of PSIP led to successful improvements in team effectiveness and efficiency.
One of the core missions of the Department of Energy (DOE) is to move beyond current high performance computing (HPC) capabilities toward a capable exascale computing ecosystem that accelerates scientific discovery and addresses critical challenges in energy and national security. The very nature of this mission has drawn a wide range of talented and successful scientists to work together in new ways to push beyond the status-quo toward this goal. For many scientists, their past success was achieved through efficient and agile collaboration within small trusted teams that rapidly innovate, prototype, and deliver. Thus, a key challenge for the ECP (Exascale Computing Project) is to scale this efficiency and innovation from small teams to aggregate teams of teams. While scaling agile collaboration from small teams to teams of teams may seem like a trivial transition, the path to exascale introduces significant uncertainty in HPC scientific software development for future modeling and simulation, and can cause unforeseen disruptions or inefficiencies that impede organizational productivity and innovation critical to achieving an integrated exascale vision. This paper identifies key challenges in scaling to a team of teams approach and recommends strategies for addressing them. The scientific community will take away lessons learned and recommended best practices from examples for enhancing productivity and innovation at scale for immediate use in modeling and simulation software engineering projects and programs.
Cross Reality (XR) immersive environments offer challenges and opportunities in designing for cognitive aspects (e.g. learning, memory, attention, etc.) of information design and interactions. Information design is a multidisciplinary endeavor involving data science, communication science, cognitive science, media, and technology. In the present paper the holodeck metaphor is extended to illustrate how information design practices and some of the qualities of this imaginary computationally augmented environment (a.k.a. the holodeck) may be achieved in XR environments to support information-rich storytelling and real life, face-to-face, and virtual collaborative interactions. The Simulation Experience Design Framework & Method is introduced to organize challenges and opportunities in the design of information for XR. The notion of carefully blending both real and virtual spaces to achieve total immersion is discussed as the reader moves through the elements of the cyclical framework. A solution space leveraging cognitive science, information design, and transmedia learning highlights key challenges facing contemporary XR designers. Challenges include but are not limited to interleaving information, technology, and media into the human storytelling process, and supporting narratives in a way that is memorable, robust, and extendable.
Marques, Osni A.; Bernholdt, David E.; Raybourn, Elaine M.; Barker, Ashley D.; Hartman-Baker, Rebecca J.
Here, in this contribution, we discuss our experiences organizing the Best Practices for HPC Software Developers (HPC-BP) webinar series, an effort for the dissemination of software development methodologies, tools and experiences to improve developer productivity and software sustainability. HPC-BP is an outreach component of the IDEAS Productivity Project [4] and has been designed to support the IDEAS mission to work with scientific software development teams to enhance their productivity and the sustainability of their codes. The series, which was launched in 2016, has just presented its 22nd webinar. We summarize and distill our experiences with these webinars, including what we consider to be "best practices" in the execution of both individual webinars and a long-running series like HPC-BP. We also discuss future opportunities and challenges in continuing the series.
Large-scale collaborative scientific software projects require more knowledge than any one person typically possesses. This makes coordination and communication of knowledge and expertise a key factor in creating and safeguarding software quality, without which we cannot have sustainable software. However, as researchers attempt to scale up the production of software, they are confronted by problems of awareness and understanding. This presents an opportunity to develop better practices and tools that directly address these challenges. To that end, we conducted a case study of developers of the Trilinos project. We surveyed the software development challenges addressed and show how those problems are connected with what they know and how they communicate. Based on these data, we provide a series of practicable recommendations, and outline a path forward for future research.
Sandia National Laboratories performed a 6-month effort to stand up a "zero-entry" cyber range environment for the purpose of providing self-directed practice to augment transmedia learning across diverse media and/or devices that may be part of a loosely coupled, distributed ecosystem. This 6-month effort leveraged Minimega, an open-source Emulytics™ (emulation + analytics) tool for launching and managing virtual machines in a cyber range. The proof of concept addressed a set of learning objectives for cybersecurity operations by providing three, short "zero-entry" exercises for beginner, intermediate, and advanced levels in network forensics, social engineering, penetration testing, and reverse engineering. Learners provided answers to problems they explored in networked virtual machines. The hands-on environment, Cyber Scorpion, participated in a preliminary demonstration in April 2017 at Ft. Bragg, NC. The present chapter describes the learning experience research and software development effort for a cybersecurity use case and subsequent lessons learned. It offers general recommendations for challenges which may be present in future learning ecosystems.
Mixed, augmented, and virtual reality holds promise for many securityrelated applications including physical security systems. When combined with models of a site, an augmented reality (AR) approach can be designed to enhance knowledge and understanding of the status of the facility. The present chapter describes how improved modeling and simulation will increase situational awareness by blurring the lines among the use of tools for analysis, rehearsal, and training-especially when coupled with immersive interaction experiences offered by augmented reality. We demonstrate how the notion of a digital twin can blur these lines. We conclude with challenges that must be overcome when applying digital twins, advanced modeling, and augmented reality to the design and development of next-generation physical security systems.
Raybourn, Elaine M.; Raybourn, Elaine M.; Schatz, Sae S.; Schatz, Sae S.; Volgol-Walcutt, Jennifer V.; Volgol-Walcutt, Jennifer V.; Vierling, Kendy V.; Vierling, Kendy V.
In this study, the authors define a new approach that addresses the challenge of efficiently designing informational artefacts for optimal knowledge acquisition, an important issue in cognitive ergonomics. Termed Industrial Knowledge Design (or InK'D), it draws from information-related (e.g. informatics) and neurosciences-related (e.g. neuroergonomics) disciplines. Although it can be used for a broad scope of communication-driven business functions, our focus as learning professionals is on conveying knowledge for purposes of training, education, and performance support. This paper discusses preliminary principles of InK'D practice that can be employed to maximise the quality and quantity of transferred knowledge through interaction design. The paper codifies tacit knowledge into explicit concepts that can be leveraged by expert and non-expert knowledge designers alike.
Next generation learning ecosystems will be comprised of intelligent, adaptive environments that utilize one’s cultural footprints to co-create shared narratives and facilitate intercultural understanding. The present paper discusses why digital footprints, cultural signposts, intercultural agents, and transmedia learning are needed to realize relevant learning in virtual environments. The paper introduces notions that may impact the design of culturally-aware information technology for distributed learning are presented.
People use social media resources like Twitter, Facebook, forums etc. to shareand discuss various activities or topics. By aggregating topic trends acrossmany individuals using these services, we seek to construct a richer profileof a person's activities and interests as well as provide a broader context ofthose activities. This profile may then be used in a variety of ways tounderstand groups as a collection of interests and affinities and anindividual's participation in those groups. Our approach considers that muchof these data will be unstructured, free-form text. By analyzing free-form text directly, we may be able to gain an implicit grouping ofindividuals with shared interests based on shared conversation, and not onexplicit social software linking them. In this paper, we discuss aproof-of-concept application called Grandmaster built to pull short sections oftext, a person's comments or Twitter posts, together by analysis andvisualization to allow a gestalt understanding of the full collection of allindividuals: how groups are similar and how they differ, based on theirtext inputs.
People use social media resources like Twitter, Facebook, forums etc. to share and discuss various activities or topics. By aggregating topic trends across many individuals using these services, we seek to construct a richer profile of a person’s activities and interests as well as provide a broader context of those activities. This profile may then be used in a variety of ways to understand groups as a collection of interests and affinities and an individual’s participation in those groups. Our approach considers that much of these data will be unstructured, free-form text. By analyzing free-form text directly, we may be able to gain an implicit grouping of individuals with shared interests based on shared conversation, and not on explicit social software linking them. In this paper, we discuss a proof-of-concept application called Grandmaster built to pull short sections of text, a person’s comments or Twitter posts, together by analysis and visualization to allow a gestalt understanding of the full collection of all individuals: how groups are similar and how they differ, based on their text inputs.
A hypothetical scenario is utilized to explore privacy and security considerations for intelligent systems, such as a Personal Assistant for Learning (PAL). Two categories of potential concerns are addressed: factors facilitated by user models, and factors facilitated by systems. Among the strategies presented for risk mitigation is a call for ongoing, iterative dialog among privacy, security, and personalization researchers during all stages of development, testing, and deployment.
International Defense and Homeland Security Simulation Workshop, DHSS 2011, Held at the International Mediterranean and Latin American Modeling Multiconference, I3M 2011
In the present paper the act of learner reflection during training with an adaptive or predictive computer-based tutor is considered a learner-system interaction. Incorporating reflection and real-time evaluation of peer performance into adaptive and predictive computerbased tutoring can support the development of automated adaptation. Allowing learners to refine and inform student models from reflective practice with independent open learner models may improve overall accuracy and relevancy. Given the emphasis on selfdirected peer learning with adaptive technology, learner and instructor modeling research continue to be critical research areas for education and training technology.
Complex problem solving approaches and novel strategies employed by the military at the squad, team, and commander level are often best learned experimentally. Since live action exercises can be costly, advances in simulation game training technology offer exciting ways to enhance current training. Computer games provide an environment for active, critical learning. Games open up possibilities for simultaneous learning on multiple levels; players may learn from contextual information embedded in the dynamics of the game, the organic process generated by the game, and through the risks, benefits, costs, outcomes, and rewards of alternative strategies that result from decision making. In the present paper we discuss a multiplayer computer game simulation created for the Adaptive Thinking & Leadership (ATL) Program to train Special Forces Team Leaders. The ATL training simulation consists of a scripted single-player and an immersive multiplayer environment for classroom use which leverages immersive computer game technology. We define adaptive thinking as consisting of competencies such as negotiation and consensus building skills, the ability to communicate effectively, analyze ambiguous situations, be self-aware, think innovatively, and critically use effective problem solving skills. Each of these competencies is an essential element of leader development training for the U.S. Army Special Forces. The ATL simulation is used to augment experiential learning in the curriculum for the U.S. Army JFK Special Warfare Center & School (SWCS) course in Adaptive Thinking & Leadership. The school is incorporating the ATL simulation game into two additional training pipelines (PSYOPS and Civil Affairs Qualification Courses) that are also concerned with developing cultural awareness, interpersonal communication adaptability, and rapport-building skills. In the present paper, we discuss the design, development, and deployment of the training simulation, and emphasize how the multiplayer simulation game is successfully used in the Special Forces Officer training program.
The present ITSE journal special issue on 'Learning About Social Interaction through Gaming' is the result of an invitation to the attendees of a one-day workshop on 'Social Learning Through Gaming' co-organized by the guest editors and held at the Human Factors in Computing Systems (CHI) conference on April 26, 2004 in Vienna, Austria. CHI is one of the premiere conferences on human-computer interaction. CHI 2004 attracted hundreds of delegates from all over the world. The CHI workshop program results from a competitive selection process. The Social Learning through Gaming workshop was filled to capacity and attended by approximately 25 participants from Europe and North America who submitted position papers that were refereed and selected for participation based on the relevancy and innovativeness of the research. The participants came together to share research on play, learning, games, interactive technologies, and what playing and designing games can teach us about social behaviors. The present special issue focuses on learning about social aspects through gaming: learning to socialize through games and learning games through social behavior.
The work reported in this document involves a development effort to provide combat commanders and systems engineers with a capability to explore and optimize system concepts that include operational concepts as part of the design effort. An infrastructure and analytic framework has been designed and partially developed that meets a gap in systems engineering design for combat related complex systems. The system consists of three major components: The first component consists of a design environment that permits the combat commander to perform 'what-if' types of analyses in which parts of a course of action (COA) can be automated by generic system constructs. The second component consists of suites of optimization tools designed to integrate into the analytical architecture to explore the massive design space of an integrated design and operational space. These optimization tools have been selected for their utility in requirements development and operational concept development. The third component involves the design of a modeling paradigm for the complex system that takes advantage of functional definitions and the coupled state space representations, generic measures of effectiveness and performance, and a number of modeling constructs to maximize the efficiency of computer simulations. The system architecture has been developed to allow for a future extension in which the operational concept development aspects can be performed in a co-evolutionary process to ensure the most robust designs may be gleaned from the design space(s).
In exploring the question of how humans reason in ambiguous situations or in the absence of complete information, we stumbled onto a body of knowledge that addresses issues beyond the original scope of our effort. We have begun to understand the importance that philosophy, in particular the work of C. S. Peirce, plays in developing models of human cognition and of information theory in general. We have a foundation that can serve as a basis for further studies in cognition and decision making. Peircean philosophy provides a foundation for understanding human reasoning and capturing behavioral characteristics of decision makers due to cultural, physiological, and psychological effects. The present paper describes this philosophical approach to understanding the underpinnings of human reasoning. We present the work of C. S. Peirce, and define sets of fundamental reasoning behavior that would be captured in the mathematical constructs of these newer technologies and would be able to interact in an agent type framework. Further, we propose the adoption of a hybrid reasoning model based on his work for future computational representations or emulations of human cognition.
This report documents an exploratory FY 00 LDRD project that sought to demonstrate the first steps toward a realistic computational representation of the variability encountered in individual human behavior. Realism, as conceptualized in this project, required that the human representation address the underlying psychological, cultural, physiological, and environmental stressors. The present report outlines the researchers' approach to representing cognitive, cultural, and physiological variability of an individual in an ambiguous situation while faced with a high-consequence decision that would greatly impact subsequent events. The present project was framed around a sensor-shooter scenario as a soldier interacts with an unexpected target (two young Iraqi girls). A software model of the ''Sensor Shooter'' scenario from Desert Storm was developed in which the framework consisted of a computational instantiation of Recognition Primed Decision Making in the context of a Naturalistic Decision Making model [1]. Recognition Primed Decision Making was augmented with an underlying foundation based on our current understanding of human neurophysiology and its relationship to human cognitive processes. While the Gulf War scenario that constitutes the framework for the Sensor Shooter prototype is highly specific, the human decision architecture and the subsequent simulation are applicable to other problems similar in concept, intensity, and degree of uncertainty. The goal was to provide initial steps toward a computational representation of human variability in cultural, cognitive, and physiological state in order to attain a better understanding of the full depth of human decision-making processes in the context of ambiguity, novelty, and heightened arousal.