There are many statistical challenges in the design and analysis of margin testing for product qualification. To further complicate issues, there are multiple types of margins that can be considered and there are often competing experimental designs to evaluate the various types of margin. There are two major variants of margin that must be addressed for engineered components: performance margin and design margin. They can be differentiated by the specific regions of the requirements space that they address. Performance margin are evaluated within the region where all inputs and environments are within requirements, and it expresses the difference between actual performance and the required performance of the system or component. Design margin expresses the difference between the maximum (or minimum) inputs and environments where the component continues to operate as intended (i.e. all performance requirements are still met), and the required inputs and conditions. The model Performance = f(Inputs, Environments? + ϵ (1) can be used to help frame the overall set of margin questions. The interdependence of inputs, environments, and outputs should be considered during the course of development in order to identify a complete test program that addresses both performance margin and design margin questions. Statistical methods can be utilized to produce a holistic and efficient program, both for qualitative activities that are designed to reveal margin limiters and for activities where margin quantification is desired. This paper discusses a holistic framework and taxonomy for margin testing and identifies key statistical challenges that may arise in developing such a program.
In September of 2009, a Tri-Lab team was formed to develop a set of metrics relating to the NNSA nuclear weapon surveillance program. The purpose of the metrics was to develop a more quantitative and/or qualitative metric(s) describing the results of realized or non-realized surveillance activities on our confidence in reporting reliability and assessing the stockpile. As a part of this effort, a statistical sub-team investigated various techniques and developed a complementary set of statistical metrics that could serve as a foundation for characterizing aspects of meeting the surveillance program objectives. The metrics are a combination of tolerance limit calculations and power calculations, intending to answer level-of-confidence type questions with respect to the ability to detect certain undesirable behaviors (catastrophic defects, margin insufficiency defects, and deviations from a model). Note that the metrics are not intended to gauge product performance but instead the adequacy of surveillance. This report gives a short description of four metrics types that were explored and the results of a sensitivity study conducted to investigate their behavior for various inputs. The results of the sensitivity study can be used to set the risk parameters that specify the level of stockpile problem that the surveillance program should be addressing.