Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System

Sheng Di, Hanqi Guo, Eric Pershey, Marc Snir, Franck Cappello

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

An in-depth understanding of the failure features of HPC jobs in a supercomputer is critical to the large-scale system maintenance and improvement of the service quality for users. In this paper, we investigate the features of hundreds of thousands of jobs in one of the most powerful supercomputers, the IBM Blue Gene/Q Mira, based on 2001 days of observations with a total of over 32.44 billion core-hours. We study the impact of the system's events on the jobs' execution in order to understand the system's reliability from the perspective of jobs and users. The characterization involves a joint analysis based on multiple data sources, including the reliability, availability, and serviceability (RAS) log; job scheduling log; the log regarding each job's physical execution tasks; and the I/O behavior log. We present 22 valuable takeaways based on our in-depth analysis. For instance, 99,245 job failures are reported in the job-scheduling log, a large majority (99.4%) of which are due to user behavior (such as bugs in code, wrong configuration, or misoperations). The job failures are correlated with multiple metrics and attributes, such as users/projects and job execution structure (number of tasks, scale, and core-hours). The best-fitting distributions of a failed job's execution length (or interruption interval) include Weibull, Pareto, inverse Gaussian, and Erlang/exponential, depending on the types of errors (i.e., exit codes). The RAS events affecting job executions exhibit a high correlation with users and core-hours and have a strong locality feature. In terms of the failed jobs, our similarity-based event-filtering analysis indicates that the mean time to interruption is about 3.5 days.

Original languageEnglish (US)
Title of host publicationProceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages473-484
Number of pages12
ISBN (Electronic)9781728100562
DOIs
StatePublished - Jun 2019
Event49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019 - Portland, United States
Duration: Jun 24 2019Jun 27 2019

Publication series

NameProceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019

Conference

Conference49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019
CountryUnited States
CityPortland
Period6/24/196/27/19

Fingerprint

Supercomputers
Scheduling
Availability
Large scale systems
Genes

Keywords

  • Failure Analysis
  • Fault Tolerance
  • IBM BlueGene/Q System
  • Supercomputer

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Safety, Risk, Reliability and Quality
  • Hardware and Architecture

Cite this

Di, S., Guo, H., Pershey, E., Snir, M., & Cappello, F. (2019). Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System. In Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019 (pp. 473-484). [8809553] (Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DSN.2019.00055

Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System. / Di, Sheng; Guo, Hanqi; Pershey, Eric; Snir, Marc; Cappello, Franck.

Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 473-484 8809553 (Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Di, S, Guo, H, Pershey, E, Snir, M & Cappello, F 2019, Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System. in Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019., 8809553, Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019, Institute of Electrical and Electronics Engineers Inc., pp. 473-484, 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019, Portland, United States, 6/24/19. https://doi.org/10.1109/DSN.2019.00055
Di S, Guo H, Pershey E, Snir M, Cappello F. Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System. In Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 473-484. 8809553. (Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019). https://doi.org/10.1109/DSN.2019.00055
Di, Sheng ; Guo, Hanqi ; Pershey, Eric ; Snir, Marc ; Cappello, Franck. / Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System. Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 473-484 (Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019).
@inproceedings{4b665cc68fa540479865f323c36809fa,
title = "Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System",
abstract = "An in-depth understanding of the failure features of HPC jobs in a supercomputer is critical to the large-scale system maintenance and improvement of the service quality for users. In this paper, we investigate the features of hundreds of thousands of jobs in one of the most powerful supercomputers, the IBM Blue Gene/Q Mira, based on 2001 days of observations with a total of over 32.44 billion core-hours. We study the impact of the system's events on the jobs' execution in order to understand the system's reliability from the perspective of jobs and users. The characterization involves a joint analysis based on multiple data sources, including the reliability, availability, and serviceability (RAS) log; job scheduling log; the log regarding each job's physical execution tasks; and the I/O behavior log. We present 22 valuable takeaways based on our in-depth analysis. For instance, 99,245 job failures are reported in the job-scheduling log, a large majority (99.4{\%}) of which are due to user behavior (such as bugs in code, wrong configuration, or misoperations). The job failures are correlated with multiple metrics and attributes, such as users/projects and job execution structure (number of tasks, scale, and core-hours). The best-fitting distributions of a failed job's execution length (or interruption interval) include Weibull, Pareto, inverse Gaussian, and Erlang/exponential, depending on the types of errors (i.e., exit codes). The RAS events affecting job executions exhibit a high correlation with users and core-hours and have a strong locality feature. In terms of the failed jobs, our similarity-based event-filtering analysis indicates that the mean time to interruption is about 3.5 days.",
keywords = "Failure Analysis, Fault Tolerance, IBM BlueGene/Q System, Supercomputer",
author = "Sheng Di and Hanqi Guo and Eric Pershey and Marc Snir and Franck Cappello",
year = "2019",
month = "6",
doi = "10.1109/DSN.2019.00055",
language = "English (US)",
series = "Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
pages = "473--484",
booktitle = "Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019",
address = "United States",

}

TY - GEN

T1 - Characterizing and Understanding HPC Job Failures over the 2K-Day Life of IBM BlueGene/Q System

AU - Di, Sheng

AU - Guo, Hanqi

AU - Pershey, Eric

AU - Snir, Marc

AU - Cappello, Franck

PY - 2019/6

Y1 - 2019/6

N2 - An in-depth understanding of the failure features of HPC jobs in a supercomputer is critical to the large-scale system maintenance and improvement of the service quality for users. In this paper, we investigate the features of hundreds of thousands of jobs in one of the most powerful supercomputers, the IBM Blue Gene/Q Mira, based on 2001 days of observations with a total of over 32.44 billion core-hours. We study the impact of the system's events on the jobs' execution in order to understand the system's reliability from the perspective of jobs and users. The characterization involves a joint analysis based on multiple data sources, including the reliability, availability, and serviceability (RAS) log; job scheduling log; the log regarding each job's physical execution tasks; and the I/O behavior log. We present 22 valuable takeaways based on our in-depth analysis. For instance, 99,245 job failures are reported in the job-scheduling log, a large majority (99.4%) of which are due to user behavior (such as bugs in code, wrong configuration, or misoperations). The job failures are correlated with multiple metrics and attributes, such as users/projects and job execution structure (number of tasks, scale, and core-hours). The best-fitting distributions of a failed job's execution length (or interruption interval) include Weibull, Pareto, inverse Gaussian, and Erlang/exponential, depending on the types of errors (i.e., exit codes). The RAS events affecting job executions exhibit a high correlation with users and core-hours and have a strong locality feature. In terms of the failed jobs, our similarity-based event-filtering analysis indicates that the mean time to interruption is about 3.5 days.

AB - An in-depth understanding of the failure features of HPC jobs in a supercomputer is critical to the large-scale system maintenance and improvement of the service quality for users. In this paper, we investigate the features of hundreds of thousands of jobs in one of the most powerful supercomputers, the IBM Blue Gene/Q Mira, based on 2001 days of observations with a total of over 32.44 billion core-hours. We study the impact of the system's events on the jobs' execution in order to understand the system's reliability from the perspective of jobs and users. The characterization involves a joint analysis based on multiple data sources, including the reliability, availability, and serviceability (RAS) log; job scheduling log; the log regarding each job's physical execution tasks; and the I/O behavior log. We present 22 valuable takeaways based on our in-depth analysis. For instance, 99,245 job failures are reported in the job-scheduling log, a large majority (99.4%) of which are due to user behavior (such as bugs in code, wrong configuration, or misoperations). The job failures are correlated with multiple metrics and attributes, such as users/projects and job execution structure (number of tasks, scale, and core-hours). The best-fitting distributions of a failed job's execution length (or interruption interval) include Weibull, Pareto, inverse Gaussian, and Erlang/exponential, depending on the types of errors (i.e., exit codes). The RAS events affecting job executions exhibit a high correlation with users and core-hours and have a strong locality feature. In terms of the failed jobs, our similarity-based event-filtering analysis indicates that the mean time to interruption is about 3.5 days.

KW - Failure Analysis

KW - Fault Tolerance

KW - IBM BlueGene/Q System

KW - Supercomputer

UR - http://www.scopus.com/inward/record.url?scp=85072119809&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85072119809&partnerID=8YFLogxK

U2 - 10.1109/DSN.2019.00055

DO - 10.1109/DSN.2019.00055

M3 - Conference contribution

AN - SCOPUS:85072119809

T3 - Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019

SP - 473

EP - 484

BT - Proceedings - 49th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2019

PB - Institute of Electrical and Electronics Engineers Inc.

ER -