The growth of High Performance Computer (HPC) systems increases the complexity with respect to understanding resource utilization, system management, and performance issues. While raw performance data is increasingly exposed at the component level, the usefulness of the data is dependent on the ability to do meaningful analysis on actionable timescales. However, current system monitoring infrastructures largely focus on data collection, with analysis performed off-system in post-processing mode. This increases the time required to provide analysis and feedback to a variety of consumers. In this work, we enhance the architecture of a monitoring system used on large-scale computational platforms, to integrate streaming analysis capabilities at arbitrary locations within its data collection, transport, and aggregation facilities. We leverage the flexible communication topology of the monitoring system to enable placement of transformations based on overhead concerns, while still enabling low-latency exposure on node. Our design internally supports and exposes the raw and transformed data uniformly for both node level and off-system consumers. We show the viability of our implementation for a case with production-relevance: run-time determination of the relative per-node files system demands.
Ahlgren, Ville A.; Andersson, Stefan A.; Brandt, James M.; Cardo, Nicholas C.; Chunduri, Sudheer C.; Enos, Jeremy E.; Fields, Parks F.; Gentile, Ann C.; Gerber, Richard G.; Greenseid, Joe G.; Greiner, Annette G.; Hadri, Bilel H.; He, Yun H.; Hoppe, Dennis H.; Kaila, Urpo K.; Kelly, Kaki K.; Klein, Mark K.; Kristiansen, Alex K.; Leak, Steve L.; Mason, Mike M.; Pedretti, Kevin P.; Piccinali, Jean-Guillaume P.; Repik, Jason; Rogers, Jim R.; Salminen, Susanna S.; Showerman, Mike S.; Whitney, Cary W.; Williams, Jim W.
Ahlgren, Ville A.; Andersson, Stefan A.; Brandt, James M.; Cardo, Nicholas C.; Chunduri, Sudheer C.; Enos, Jeremy E.; Fields, Parks F.; Gentile, Ann C.; Gerber, Richard G.; Greenseid, Joe G.; Greiner, Annette G.; Hadri, Bilel H.; He, Yun H.; Hoppe, Dennis H.; Kaila, Urpo K.; Kelly, Kaki K.; Klein, Mark K.; Kristiansen, Alex K.; Leak, Steve L.; Mason, Mike M.; Pedretti, Kevin P.; Piccinali, Jean-Guillaume P.; Repik, Jason; Rogers, Jim R.; Salminen, Susanna S.; Showerman, Mike S.; Whitney, Cary W.; Williams, Jim W.
Proceedings - IEEE International Conference on Cluster Computing, ICCC
Jha, Saurabh; Brandt, James M.; Gentile, Ann C.; Kalbarczyk, Zbigniew; Bauer, Greg; Enos, Jeremy; Showerman, Michael; Kaplan, Larry; Bode, Brett; Greiner, Annette; Bonnie, Amanda; Mason, Mike; Iyer, Ravishankar K.; Kramer, William
In high-performance computing systems, application performance and throughput are dependent on a complex interplay of hardware and software subsystems and variable workloads with competing resource demands. Data-driven insights into the potentially widespread scope and propagationof impact of events, such as faults and contention for shared resources, can be used to drive more effective use of resources, for improved root cause diagnosis, and for predicting performance impacts. We present work developing integrated capabilities for holistic monitoring and analysis to understand and characterize propagation of performance-degrading events. These characterizations can be used to determine and invoke mitigating responses by system administrators, applications, and system software.
The presentation documented the technical approach of the team and summary of the results with sufficient detail to demonstrate both the value and the completion of the milestone. A separate SAND report was also generated with more detail to supplement the presentation.
Jha, Saurabh J.; Brandt, James M.; Gentile, Ann C.; Karlbarczyk, Zbigniew K.; Bauer, Greg B.; Enos, Jeremy E.; Showerman, Michael S.; Kaplan, Larry K.; Bode, Brett B.; Greiner, Annette G.; Bonnie, Amanda B.; Mason, Mike M.; Iyer, Ravishankar I.; Kramer, William K.
Formicola, Valerio F.; Jha, Saurabh J.; Chen, Daniel C.; Dong, Wen D.; Bonnie, Amanda B.; Mason, Mike M.; Brandt, James M.; Gentile, Ann C.; Kaplan, Larry K.; Repik, Jason; Enos, Jeremy E.; Showerman, Mike S.; Greiner, Annette G.; Kalbarczyk, Zbigniew K.; Iyer, Ravishankar I.; Kramer, Bill K.
A detailed understanding of HPC applications’ resource needs and their complex interactions with each other and HPC platform resources are critical to achieving scalability and performance. Such understanding has been difficult to achieve because typical application profiling tools do not capture the behaviors of codes under the potentially wide spectrum of actual production conditions and because typical monitoring tools do not capture system resource usage information with high enough fidelity to gain sufficient insight into application performance and demands. In this paper we present both system and application profiling results based on data obtained through synchronized system wide monitoring on a production HPC cluster at Sandia National Laboratories (SNL). We demonstrate analytic and visualization techniques that we are using to characterize application and system resource usage under production conditions for better understanding of application resource needs. Our goals are to improve application performance (through understanding application-to-resource mapping and system throughput) and to ensure that future system capabilities match their intended workloads.