JETS: Language and System Support for Many-Parallel-Task Workflows

Justin M. Wozniak, Michael Wilde, Daniel S. Katz

Research output: Contribution to journalArticlepeer-review

Abstract

Many-task computing is a well-established paradigm for implementing loosely coupled applications (tasks) on large-scale computing systems. However, few of the model's existing implementations provide efficient, low-latency support for executing tasks that are tightly coupled multiprocessing applications. Thus, a vast array of parallel applications cannot readily be used effectively within many-task workloads. In this work, we present JETS, a middleware component that provides high performance support for many-parallel-task computing (MPTC). JETS is based on a highly concurrent approach to parallel task dispatch and on new capabilities now available in the MPICH2 MPI implementation and the ZeptoOS Linux operating system. JETS represents an advance over the few known examples of multilevel many-parallel-task scheduling systems: it more efficiently schedules and launches many short-duration parallel application invocations; it overcomes the challenges of coupling the user processes of each multiprocessing application invocation via the messaging fabric; and it concurrently manages many application executions in various stages. We report here on the JETS architecture and its performance on both synthetic benchmarks and an MPTC application in molecular dynamics.

Original languageEnglish (US)
Pages (from-to)341-360
Number of pages20
JournalJournal of Grid Computing
Volume11
Issue number3
DOIs
StatePublished - Sep 2013
Externally publishedYes

Keywords

  • JETS
  • MPI
  • MPTC
  • MTC
  • NAMD
  • Swift
  • Workflow

ASJC Scopus subject areas

  • Software
  • Information Systems
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'JETS: Language and System Support for Many-Parallel-Task Workflows'. Together they form a unique fingerprint.

Cite this