Measuring and Understanding Extreme-Scale Application Resilience: A Field Study of 5,000,000 HPC Application Runs

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

This paper presents an in-depth characterization of the resiliency of more than 5 million HPC application runs completed during the first 518 production days of Blue Waters, a 13.1 petaflop Cray hybrid supercomputer. Unlike past work, we measure the impact of system errors and failures on user applications, i.e., the compiled programs launched by user jobs that can execute across one or more XE (CPU) or XK (CPU+GPU) nodes. The characterization is performed by means of a joint analysis of several data sources, which include workload and error/failure logs. In order to relate system errors and failures to the executed applications, we developed LogDiver, a tool to automate the data pre-processing and metric computation. Some of the lessons learned in this study include: i) while about 1.53% of applications fail due to system problems, the failed applications contribute to about 9% of the production node hours executed in the measured period, i.e., the system consumes computing resources, and system-related issues represent a potentially significant energy cost for the work lost, ii) there is a dramatic increase in the application failure probability when executing full-scale applications: 20x (from 0.008 to 0.162) when scaling XE applications from 10,000 to 22,000 nodes, and 6x (from 0.02 to 0.129) when scaling GPU/hybrid applications from 2000 to 4224 nodes, and iii) the resiliency of hybrid applications is impaired by the lack of adequate error detection capabilities in hybrid nodes.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2015
PublisherIEEE Computer Society
Pages25-36
Number of pages12
ISBN (Electronic)9781479986293
DOIs
StatePublished - Sep 14 2015
Event45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2015 - Rio de Janeiro, Brazil
Duration: Jun 22 2015Jun 25 2015

Publication series

NameProceedings of the International Conference on Dependable Systems and Networks
Volume2015-September

Other

Other45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2015
CountryBrazil
CityRio de Janeiro
Period6/22/156/25/15

Keywords

  • application resilience
  • data analysis
  • data-driven resilience
  • extreme-scale
  • hybrid machines
  • resilience
  • supercomputer

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Measuring and Understanding Extreme-Scale Application Resilience: A Field Study of 5,000,000 HPC Application Runs'. Together they form a unique fingerprint.

Cite this