A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications

Suraj Prabhakaran, Marcel Neumann, Sebastian Rinke, Felix Wolf, Abhishek Gupta, Laxmikant V Kale

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The throughput of supercomputers depends not only on efficient job scheduling but also on the type of jobs that form the workload. Malleable jobs are most favourable for a cluster as they can dynamically adapt to a changing allocation of resources. The batch system can expand or shrink a running malleable job to improve system utilization, throughput, and response times. In the past, however, the rigid nature of commonly used programming models like MPI made writing malleable applications a daunting task, which is why it remained largely unrealized. This is now changing. To improve fault tolerance, load imbalance, and energy efficiency in emerging exactable systems, more adaptive programming paradigms such as Charm++ enter the scene. Although they offer better support for malleability, current batch systems still lack management facilities for malleable jobs and are therefore incapable of leveraging their potential. In this paper, we present an extension of the Torque/Maui batch system for malleability. We propose a novel malleable job scheduling strategy and show the first batch system capable of efficiently managing rigid, malleable, and evolving jobs together. We demonstrate that our strategy achieves consistently superior performance in comparison to every other state-of-the-art malleable job scheduling strategy under varying dynamics of the workload.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages429-438
Number of pages10
ISBN (Electronic)9781479986484
DOIs
StatePublished - Jul 17 2015
Event29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015 - Hyderabad, India
Duration: May 25 2015May 29 2015

Publication series

NameProceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015

Other

Other29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015
CountryIndia
CityHyderabad
Period5/25/155/29/15

Keywords

  • adaptive resource management
  • adaptive scheduling
  • batch systems
  • evolving jobs
  • malleable jobs

ASJC Scopus subject areas

  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications'. Together they form a unique fingerprint.

  • Cite this

    Prabhakaran, S., Neumann, M., Rinke, S., Wolf, F., Gupta, A., & Kale, L. V. (2015). A Batch System with Efficient Adaptive Scheduling for Malleable and Evolving Applications. In Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015 (pp. 429-438). [7161531] (Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2015.34