Parsl: Scalable parallel scripting in python

Yadu Babuji, Kyle Chard, Ian Foster, Daniel S. Katz, Michael Wilde, Anna Woodard, Justin Wozniak

Research output: Contribution to journalConference article

Abstract

Computational and data-driven research practices have significantly changed over the past decade to encompass new analysis models such as interactive and online computing. Science gateways are simultaneously evolving to support this transforming landscape with the aim to enable transparent, scalable execution of a variety of analyses. Science gateways often rely on workflow management systems to represent and execute analyses efficiently and reliably. However, integrating workflow systems in science gateways can be challenging, especially as analyses become more interactive and dynamic, requiring sophisticated orchestration and management of applications and data, and customization for specific execution environments. Parsl (Parallel Scripting Library), a Python library for programming and executing data-oriented workflows in parallel, addresses these problems. Developers simply annotate a Python script with Parsl directives wrapping either Python functions or calls to external applications. Parsl manages the execution of the script on clusters, clouds, grids, and other resources; orchestrates required data movement; and manages the execution of Python functions and external applications in parallel. The Parsl library can be easily integrated into Python-based gateways, allowing for simple management and scaling of workflows.

Original languageEnglish (US)
JournalCEUR Workshop Proceedings
Volume2357
StatePublished - Jan 1 2019
Event10th International Workshop on Science Gateways, IWSG 2018 - Edinburgh, United Kingdom
Duration: Jun 13 2018Jun 15 2018

Keywords

  • Parallel scripting
  • Parsl
  • Python
  • Scientific Workflows

ASJC Scopus subject areas

  • Computer Science(all)

Cite this

Babuji, Y., Chard, K., Foster, I., Katz, D. S., Wilde, M., Woodard, A., & Wozniak, J. (2019). Parsl: Scalable parallel scripting in python. CEUR Workshop Proceedings, 2357.