AME: An anyscale Many-Task Computing engine

Zhao Zhang, Daniel S. Katz, Matei Ripeanu, Michael Wilde, Ian Foster

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Many-Task Computing (MTC) is a new application category that encompasses increasingly popular applications in biology, economics, and statistics. The high inter-task parallelism and data-intensive processing capabilities of these applications pose new challenges to existing supercomputer hardware-software stacks. These challenges include resource provisioning; task dispatching, dependency resolution, and load balancing; data management; and resilience. This paper examines the characteristics of MTC applications which create these challenges, and identifies related gaps in the middleware that supports these applications on extreme-scale systems. Based on this analysis, we propose AME, an Anyscale MTC Engine, which addresses the scalability aspects of these gaps. We describe the AME framework and present performance results for both synthetic benchmarks and real applications. Our results show that AME's dispatching performance linearly scales up to 14,120 tasks/second on 16,384 cores with high efficiency. The overhead of the intermediate data management scheme does not increase significantly up to 16,384 cores. AME eliminates 73% of the file transfer between compute nodes and the global filesystem for the Montage astronomy application running on 2,048 cores. Our results indicate that AME scales well on today's petascale machines, and is a strong candidate for exascale machines.

Original languageEnglish (US)
Title of host publicationWORKS'11 - Proceedings of the 6th Workshop on Workflows in Support of Large-Scale Science, Co-located with SC'11
Pages137-146
Number of pages10
DOIs
StatePublished - 2011
Externally publishedYes
Event6th Workshop on Workflows in Support of Large-Scale Science, WORKS'11, Co-located with SC'11 - Seattle, WA, United States
Duration: Nov 14 2011Nov 14 2011

Publication series

NameWORKS'11 - Proceedings of the 6th Workshop on Workflows in Support of Large-Scale Science, Co-located with SC'11

Other

Other6th Workshop on Workflows in Support of Large-Scale Science, WORKS'11, Co-located with SC'11
CountryUnited States
CitySeattle, WA
Period11/14/1111/14/11

Keywords

  • Data management
  • Load balancing
  • Many-Task Computing
  • Scheduling
  • Supercomputer systems

ASJC Scopus subject areas

  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'AME: An anyscale Many-Task Computing engine'. Together they form a unique fingerprint.

Cite this