Scheduling many-task workloads on supercomputers: Dealing with trailing tasks

Timothy G. Armstrong, Zhao Zhang, Daniel S. Katz, Michael Wilde, Ian T. Foster

Research output: Chapter in Book/Report/Conference proceedingConference contribution


In order for many-task applications to be attractive candidates for running on high-end supercomputers, they must be able to benefit from the additional compute, I/O, and communication performance provided by high-end HPC hardware relative to clusters, grids, or clouds. Typically this means that the application should use the HPC resource in such a way that it can reduce time to solution beyond what is possible otherwise. Furthermore, it is necessary to make efficient use of the computational resources, achieving high levels of utilization. Satisfying these twin goals is not trivial, because while the parallelism in many task computations can vary over time, on many large machines the allocation policy requires that worker CPUs be provisioned and also relinquished in large blocks rather than individually. This paper discusses the problem in detail, explaining and characterizing the trade-off between utilization and time to solution under the allocation policies of Blue Gene/P Intrepid at Argonne National Laboratory. We propose and test two strategies to improve this trade-off: scheduling tasks in order of longest to shortest (applicable only if task runtimes are predictable) and downsizing allocations when utilization drops below some threshold. We show that both strategies are effective under different conditions.

Original languageEnglish (US)
Title of host publication2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS10
PublisherIEEE Computer Society
ISBN (Print)9781424497041
StatePublished - 2010
Externally publishedYes

Publication series

Name2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS10


  • High-performance computing
  • Many-task computing
  • Scheduling
  • Supercomputer systems

ASJC Scopus subject areas

  • Hardware and Architecture


Dive into the research topics of 'Scheduling many-task workloads on supercomputers: Dealing with trailing tasks'. Together they form a unique fingerprint.

Cite this