Exploring Properties and Correlations of Fatal Events in a Large-Scale HPC System

Sheng Di, Hanqi Guo, Rinku Gupta, Eric R. Pershey, Marc Snir, Franck Cappello

Research output: Contribution to journalArticle

Abstract

In this paper, we explore potential correlations of fatal system events for one of the most powerful supercomputers - IBM Blue Gene/Q Mira, which is deployed at Argonne National Laboratory, based on its 5-year reliability, availability, and serviceability (RAS) log. Our contribution is two-fold. (1) We design an efficient log analysis tool, namely LogAider, with a novel filtering method to effectively extract fatal events from masses of system messages that are heavily duplicated in the log. LogAider exhibits a very precise detection of temporal-correlation with a high similarity (up to 95 percent) to the ground-truth (i.e., compared to the failure records reported by the administrators). The total number of fatal events can be reduced to about 1,255 compared with originally 2.6 million duplicated fatal messages. (2) We analyze the 5-year RAS log of the MIRA system using LogAider, and summarize six important 'takeaways' which can help system vendors and administrators better understand an extreme-scale system's fatal events. Specifically, we find that the distribution or proportion of the fatal system events follow a Pareto-like principle in general. The temporal correlation among fatal events is much stronger than that of warn messages and info messages, and the correlated events tend to constitute a few clusters. The mean time between fatal events (MTBFE) of the Mira system is about 1.3 days from the perspective of the system, and the MTTI is 2-4 days from the perspective of users. The most error-prone item value with respect to any key attribute appears likely in the log every 2-10 days. Weibull, Gamma, and Pearson6 are the three best-fit distributions for the fatal event intervals. The overall correlation of fatal events on the 5D torus network is not prominent, whereas the small-region locality correlation (e.g., the fatal events inside racks) is relatively strong. We believe our work will be interesting to large-scale HPC system administrators and vendors and to fault tolerance researchers, enabling them to better understand fatal events and mitigate such events accordingly.

Original languageEnglish (US)
Article number8436427
Pages (from-to)361-374
Number of pages14
JournalIEEE Transactions on Parallel and Distributed Systems
Volume30
Issue number2
DOIs
StatePublished - Feb 1 2019

    Fingerprint

Keywords

  • Peta-scale supercomputer
  • fatal event analysis
  • mining correlations
  • reliability-availability-serviceability (RAS)

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics

Cite this