Scalable parallel programming in python with PArsL

Yadu Babuji, Anna Woodard, Zhuozhao Li, Daniel S Katz, Ben Clifford, Ian Foster, Michael Wilde, Kyle Chard

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Python is increasingly the lingua franca of scientific computing. It is used as a higher level language to wrap lower-level libraries and to compose scripts from various independent components. However, scaling and moving Python programs from laptops to supercomputers remains a challenge. Here we present Parsl, a parallel scripting library for Python. Parsl makes it straightforward for developers to implement parallelism in Python by annotating functions that can be executed asynchronously and in parallel, and to scale analyses from a laptop to thousands of nodes on a supercomputer or distributed system. We examine how Parsl is implemented, focusing on syntax and usage. We describe two scientific use cases in which Parsl’s intuitive and scalable parallelism is used.

Original languageEnglish (US)
Title of host publicationProceedings of the Practice and Experience in Advanced Research Computing
Subtitle of host publicationRise of the Machines (Learning), PEARC 2019
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450372275
DOIs
StatePublished - Jul 28 2019
Event2019 Conference on Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019 - Chicago, United States
Duration: Jul 28 2019Aug 1 2019

Publication series

NameACM International Conference Proceeding Series

Conference

Conference2019 Conference on Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019
CountryUnited States
CityChicago
Period7/28/198/1/19

Fingerprint

Parallel programming
Supercomputers
Natural sciences computing
High level languages

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Computer Networks and Communications
  • Computer Vision and Pattern Recognition
  • Software

Cite this

Babuji, Y., Woodard, A., Li, Z., Katz, D. S., Clifford, B., Foster, I., ... Chard, K. (2019). Scalable parallel programming in python with PArsL. In Proceedings of the Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019 [3332231] (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3332186.3332231

Scalable parallel programming in python with PArsL. / Babuji, Yadu; Woodard, Anna; Li, Zhuozhao; Katz, Daniel S; Clifford, Ben; Foster, Ian; Wilde, Michael; Chard, Kyle.

Proceedings of the Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019. Association for Computing Machinery, 2019. 3332231 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Babuji, Y, Woodard, A, Li, Z, Katz, DS, Clifford, B, Foster, I, Wilde, M & Chard, K 2019, Scalable parallel programming in python with PArsL. in Proceedings of the Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019., 3332231, ACM International Conference Proceeding Series, Association for Computing Machinery, 2019 Conference on Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019, Chicago, United States, 7/28/19. https://doi.org/10.1145/3332186.3332231
Babuji Y, Woodard A, Li Z, Katz DS, Clifford B, Foster I et al. Scalable parallel programming in python with PArsL. In Proceedings of the Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019. Association for Computing Machinery. 2019. 3332231. (ACM International Conference Proceeding Series). https://doi.org/10.1145/3332186.3332231
Babuji, Yadu ; Woodard, Anna ; Li, Zhuozhao ; Katz, Daniel S ; Clifford, Ben ; Foster, Ian ; Wilde, Michael ; Chard, Kyle. / Scalable parallel programming in python with PArsL. Proceedings of the Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019. Association for Computing Machinery, 2019. (ACM International Conference Proceeding Series).
@inproceedings{82228519759643d1953f6edb90b67941,
title = "Scalable parallel programming in python with PArsL",
abstract = "Python is increasingly the lingua franca of scientific computing. It is used as a higher level language to wrap lower-level libraries and to compose scripts from various independent components. However, scaling and moving Python programs from laptops to supercomputers remains a challenge. Here we present Parsl, a parallel scripting library for Python. Parsl makes it straightforward for developers to implement parallelism in Python by annotating functions that can be executed asynchronously and in parallel, and to scale analyses from a laptop to thousands of nodes on a supercomputer or distributed system. We examine how Parsl is implemented, focusing on syntax and usage. We describe two scientific use cases in which Parsl’s intuitive and scalable parallelism is used.",
author = "Yadu Babuji and Anna Woodard and Zhuozhao Li and Katz, {Daniel S} and Ben Clifford and Ian Foster and Michael Wilde and Kyle Chard",
year = "2019",
month = "7",
day = "28",
doi = "10.1145/3332186.3332231",
language = "English (US)",
series = "ACM International Conference Proceeding Series",
publisher = "Association for Computing Machinery",
booktitle = "Proceedings of the Practice and Experience in Advanced Research Computing",

}

TY - GEN

T1 - Scalable parallel programming in python with PArsL

AU - Babuji, Yadu

AU - Woodard, Anna

AU - Li, Zhuozhao

AU - Katz, Daniel S

AU - Clifford, Ben

AU - Foster, Ian

AU - Wilde, Michael

AU - Chard, Kyle

PY - 2019/7/28

Y1 - 2019/7/28

N2 - Python is increasingly the lingua franca of scientific computing. It is used as a higher level language to wrap lower-level libraries and to compose scripts from various independent components. However, scaling and moving Python programs from laptops to supercomputers remains a challenge. Here we present Parsl, a parallel scripting library for Python. Parsl makes it straightforward for developers to implement parallelism in Python by annotating functions that can be executed asynchronously and in parallel, and to scale analyses from a laptop to thousands of nodes on a supercomputer or distributed system. We examine how Parsl is implemented, focusing on syntax and usage. We describe two scientific use cases in which Parsl’s intuitive and scalable parallelism is used.

AB - Python is increasingly the lingua franca of scientific computing. It is used as a higher level language to wrap lower-level libraries and to compose scripts from various independent components. However, scaling and moving Python programs from laptops to supercomputers remains a challenge. Here we present Parsl, a parallel scripting library for Python. Parsl makes it straightforward for developers to implement parallelism in Python by annotating functions that can be executed asynchronously and in parallel, and to scale analyses from a laptop to thousands of nodes on a supercomputer or distributed system. We examine how Parsl is implemented, focusing on syntax and usage. We describe two scientific use cases in which Parsl’s intuitive and scalable parallelism is used.

UR - http://www.scopus.com/inward/record.url?scp=85071010264&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85071010264&partnerID=8YFLogxK

U2 - 10.1145/3332186.3332231

DO - 10.1145/3332186.3332231

M3 - Conference contribution

AN - SCOPUS:85071010264

T3 - ACM International Conference Proceeding Series

BT - Proceedings of the Practice and Experience in Advanced Research Computing

PB - Association for Computing Machinery

ER -