Optimizing I/O performance of HPC applications with autotuning

Babak Behzad, Surendra Byna, Prabhat, Marc Snir

Research output: Contribution to journalArticle

Abstract

Parallel Input output is an essential component of modern high-performance computing (HPC). Obtaining good I/O performance for a broad range of applications on diverse HPC platforms is a major challenge, in part, because of complex inter dependencies between I/O middleware and hardware. The parallel file system and I/O middleware layers all offer optimization parameters that can, in theory, result in better I/O performance. Unfortunately, the right combination of parameters is highly dependent on the application, HPC platform, problem size, and concurrency. Scientific application developers do not have the time or expertise to take on the substantial burden of identifying good parameters for each problem configuration. They resort to using system defaults, a choice that frequently results in poor I/O performance. We expect this problem to be compounded on exascale-class machines, which will likely have a deeper software stack with hierarchically arranged hardware resources. We present as a solution to this problem an autotuning system for optimizing I/O performance, I/O performance modeling, I/O tuning, and I/O patterns. We demonstrate the value of this framework across several HPC platforms and applications at scale.

Original languageEnglish (US)
Article number15
JournalACM Transactions on Parallel Computing
Volume5
Issue number4
DOIs
StatePublished - Mar 2019

Keywords

  • Autotuning
  • HPC
  • I/O
  • Parallel file systems
  • Performance optimization
  • Storage

ASJC Scopus subject areas

  • Software
  • Modeling and Simulation
  • Hardware and Architecture
  • Computer Science Applications
  • Computational Theory and Mathematics

Fingerprint Dive into the research topics of 'Optimizing I/O performance of HPC applications with autotuning'. Together they form a unique fingerprint.

  • Cite this