Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System

Sheng Di, Hanqi Guo, Eric Pershey, Marc Snir, Franck Cappello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

An in-depth understanding of the failure features of HPC jobs in a supercomputer is critical to the large-scale system maintenance and improvement of the service quality for users. In this paper, we investigate the features of hundreds of thousands of jobs in one of the most powerful supercomputers, the IBM Blue Gene/Q Mira, based on 2001 days of observations with a total of over 32.44 billion core-hours. We study the impact of the system's events on the jobs' execution in order to understand the system's reliability from the perspective of jobs and users. The characterization involves a joint analysis based on multiple data sources, including the reliability, availability, and serviceability (RAS) log; job scheduling log; the log regarding each job's physical execution tasks; and the I/O behavior log. We present 22 valuable takeaways based on our in-depth analysis. For instance, 99,245 job failures are reported in the job-scheduling log, a large majority (99.4%) of which are due to user behavior (such as bugs in code, wrong configuration, or misoperations). The job failures are correlated with multiple metrics and attributes, such as users/projects and job execution structure (number of tasks, scale, and core-hours). The best-fitting distributions of a failed job's execution length (or interruption interval) include Weibull, Pareto, inverse Gaussian, and Erlang/exponential, depending on the types of errors (i.e., exit codes). The RAS events affecting job executions exhibit a high correlation with users and core-hours and have a strong locality feature. In terms of the failed jobs, our similarity-based event-filtering analysis indicates that the mean time to interruption is about 3.5 days.

Original languageEnglish (US)
Title of host publicationProceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages473-484
Number of pages12
ISBN (Electronic)9781728100562
DOIs
StatePublished - Jun 2019
Event49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019 - Portland, United States
Duration: Jun 24 2019Jun 27 2019

Publication series

NameProceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019

Conference

Conference49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019
CountryUnited States
CityPortland
Period6/24/196/27/19

Keywords

  • Failure Analysis
  • Fault Tolerance
  • IBM BlueGene/Q System
  • Supercomputer

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Safety, Risk, Reliability and Quality
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System'. Together they form a unique fingerprint.

  • Cite this

    Di, S., Guo, H., Pershey, E., Snir, M., & Cappello, F. (2019). Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System. In Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019 (pp. 473-484). [8809553] (Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DSN.2019.00055