MTC envelope: Defining the capability of large scale computers in the context of parallel scripting applications

Zhao Zhang, Daniel S. Katz, Michael Wilde, Justin M. Wozniak, Ian Foster

Research output: Contribution to conferencePaper

Abstract

Many scientific applications can be efficiently expressed with the parallel scripting (many-task computing, MTC) paradigm. These applications are typically composed of several stages of computation, with tasks in different stages coupled by a shared file system abstraction. However, we often see poor performance when running these applications on large scale computers due to the applications' frequency and volume of filesystem I/O and the absence of appropriate optimizations in the context of parallel scripting applications. In this paper, we show the capability of existing large scale computers to run parallel scripting applications by first defining the MTC envelope and then evaluating the envelope by benchmarking a suite of shared filesystem performance metrics. We also seek to determine the origin of the performance bottleneck by profiling the parallel scripting applications' I/O behavior and mapping the I/O operations to the MTC envelope. We show an example shared filesystem envelope and present a method to predict the I/O performance given the applications' level of I/O concurrency and I/O amount. This work is instrumental in guiding the development of parallel scripting applications to make efficient use of existing large scale computers, and to evaluate performance improvements in the hardware/software stack that will better facilitate parallel scripting applications.

Original languageEnglish (US)
Pages37-48
Number of pages12
DOIs
StatePublished - Jul 17 2013
Event22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013 - New York, NY, United States
Duration: Jun 17 2013Jun 21 2013

Other

Other22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013
CountryUnited States
CityNew York, NY
Period6/17/136/21/13

Keywords

  • MTC
  • distributed file system
  • parallel scripting application
  • performance measurements

ASJC Scopus subject areas

  • Software

Fingerprint Dive into the research topics of 'MTC envelope: Defining the capability of large scale computers in the context of parallel scripting applications'. Together they form a unique fingerprint.

  • Cite this

    Zhang, Z., Katz, D. S., Wilde, M., Wozniak, J. M., & Foster, I. (2013). MTC envelope: Defining the capability of large scale computers in the context of parallel scripting applications. 37-48. Paper presented at 22nd ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2013, New York, NY, United States. https://doi.org/10.1145/2462902.2462913