Scientists and engineers at Sandia and elsewhere rely on computational simulation (CompSim) to help address a multitude of highly complex problems, from predicting weather patterns or the spread of diseases to understanding the physical forces at work in a weapons system.

One of the central challenges in developing simulations of this type revolves around a crucial question: how confident are we that the simulation is accurate?

Answering that question involves the significant task of gathering and presenting “credibility evidence,” key indicators from the simulation’s results that show fidelity to established benchmarks and expectations.

The Credibility Framework, a recently released, open-source application, streamlines that work by providing an integrated suite of tools for planning and carrying out credibility processes. 

At Sandia, a methodology for credibility processes has been in place for the nuclear enterprise for more than a decade. Key components of the methodology seek to identify and rank computational capability gaps and to focus the CompSim team’s attention on crucial facets of credibility: code verification, physics and material model fidelity, representation of geometric fidelity, solution verification, uncertainty quantification and validation.

Gathering evidence for several of these credibility “sub-elements” requires large-scale model assemblies that can only be executed on HPC platforms. For example, an uncertainty quantification study on the B61-12 weapon system took one week on Los Alamos National Laboratory’s Trinity using 120 samples each running on 1200 processors for several hours. To record and present the needed data, development teams have previously relied on an ad hoc array of spreadsheets and other electronic documents. 

The Credibility Framework (CF) application guides developers in planning for, gathering, organizing and presenting the large volumes of evidence needed to support a simulation’s reliability. The application provides direct links to analysis and ensemble workflows, and to evidence recorded by the CompSim team in CF, establishing a hierarchical, organized gateway into the detailed technical basis of credibility. The CF app also supports communication between team members, peer reviewers and customers.

The CF application represents three years’ effort by a team of Sandia engineers and software developers working with small-business partner NexGen Analytics. The application comprises a suite of tools and a standardized workflow for implementing them in doing the following:

• Planning the simulation itself. The application provides procedures and dashboards for identifying the simulation’s requirements, inputs and outputs, and an inventory of model uncertainties.
• Planning the collection of credibility evidence.
• Developing and carrying out activities that build credibility evidence. 
• Gathering the credibility evidence. 
• Reporting the credibility evidence. The application presents the top-level engineering information needed by reviewers when assessing the credibility of a computational model. 

By organizing, reporting and archiving development of credibility evidence, CF supports CompSim model readiness and other forms of peer reviews. Before CF’s public release, in a joint ASC V&V and IC level 2 milestone project, developers, analyst partners and UI/UX contributors successfully demonstrated use of the application on three nuclear deterrent (ND)-related exemplar simulations developed specifically to showcase a CompSim model development approach that produces and records credibility evidence as integral part of the modeling process.

• A range of differing mission goals and credibility evidence needs were represented by these exemplars:
• A thermal fluids simulation related to an unclassified reentry vehicle (URV). 
• A structural dynamics simulation W87-1 motivated mid bulkhead assembly.
• A structural mechanics Joint Technology Demonstrator (JTD) bolted joint.

The resulting credibility evidence packages and linked CompSim analysis artifacts are archived in the Sandia Analysis Workbench (SAW). These will assist with training new analysts and V&V practitioners, regression testing. They will also serve as realistic test beds for strategic initiatives such as the Engineering Common Modeling Framework (ECMF) and, by proxy, Accelerated Digital Engineering (ADE). 

The CF application itself is now available to any organization performing high-consequence CompSim work requiring credibility evidence. As an Eclipse plug-in, it can be integrated with SAW or operated within a generic Eclipse instance.

On-going development will provide team collaboration capabilities as well as enhancements to the reporting subsystem that generates LaTeX/PDF and MS Word report from the content of a CF project. 

In conjunction with an SDM (Simulation Data Management) system such as the one implemented in SAW, the information cataloged within CF is also useful for training and knowledge archival purposes as well as model sustainment efforts. 

CF supports current ND programs such as W80-4 and W87-1 as well as on-boarding engineers to support training them in Sandia’s credibility processes. Additionally, credibility evidence from CF is attached to analysis packages entered in the ECMF to inform long-term application of these models by archiving model credibility.