Abstract
Computational and data-driven research practices have significantly changed over the past decade to encompass new analysis models such as interactive and online computing. Science gateways are simultaneously evolving to support this transforming landscape with the aim to enable transparent, scalable execution of a variety of analyses. Science gateways often rely on workflow management systems to represent and execute analyses efficiently and reliably. However, integrating workflow systems in science gateways can be challenging, especially as analyses become more interactive and dynamic, requiring sophisticated orchestration and management of applications and data, and customization for specific execution environments. Parsl (Parallel Scripting Library), a Python library for programming and executing data-oriented workflows in parallel, addresses these problems. Developers simply annotate a Python script with Parsl directives wrapping either Python functions or calls to external applications. Parsl manages the execution of the script on clusters, clouds, grids, and other resources; orchestrates required data movement; and manages the execution of Python functions and external applications in parallel. The Parsl library can be easily integrated into Python-based gateways, allowing for simple management and scaling of workflows.
| Original language | English (US) |
|---|---|
| Journal | CEUR Workshop Proceedings |
| Volume | 2357 |
| State | Published - Jan 1 2019 |
| Event | 10th International Workshop on Science Gateways, IWSG 2018 - Edinburgh, United Kingdom Duration: Jun 13 2018 → Jun 15 2018 |
Keywords
- Parallel scripting
- Parsl
- Python
- Scientific Workflows
ASJC Scopus subject areas
- Computer Science(all)
Cite this
Parsl : Scalable parallel scripting in python. / Babuji, Yadu; Chard, Kyle; Foster, Ian; Katz, Daniel S; Wilde, Michael; Woodard, Anna; Wozniak, Justin.
In: CEUR Workshop Proceedings, Vol. 2357, 01.01.2019.Research output: Contribution to journal › Conference article
}
TY - JOUR
T1 - Parsl
T2 - Scalable parallel scripting in python
AU - Babuji, Yadu
AU - Chard, Kyle
AU - Foster, Ian
AU - Katz, Daniel S
AU - Wilde, Michael
AU - Woodard, Anna
AU - Wozniak, Justin
PY - 2019/1/1
Y1 - 2019/1/1
N2 - Computational and data-driven research practices have significantly changed over the past decade to encompass new analysis models such as interactive and online computing. Science gateways are simultaneously evolving to support this transforming landscape with the aim to enable transparent, scalable execution of a variety of analyses. Science gateways often rely on workflow management systems to represent and execute analyses efficiently and reliably. However, integrating workflow systems in science gateways can be challenging, especially as analyses become more interactive and dynamic, requiring sophisticated orchestration and management of applications and data, and customization for specific execution environments. Parsl (Parallel Scripting Library), a Python library for programming and executing data-oriented workflows in parallel, addresses these problems. Developers simply annotate a Python script with Parsl directives wrapping either Python functions or calls to external applications. Parsl manages the execution of the script on clusters, clouds, grids, and other resources; orchestrates required data movement; and manages the execution of Python functions and external applications in parallel. The Parsl library can be easily integrated into Python-based gateways, allowing for simple management and scaling of workflows.
AB - Computational and data-driven research practices have significantly changed over the past decade to encompass new analysis models such as interactive and online computing. Science gateways are simultaneously evolving to support this transforming landscape with the aim to enable transparent, scalable execution of a variety of analyses. Science gateways often rely on workflow management systems to represent and execute analyses efficiently and reliably. However, integrating workflow systems in science gateways can be challenging, especially as analyses become more interactive and dynamic, requiring sophisticated orchestration and management of applications and data, and customization for specific execution environments. Parsl (Parallel Scripting Library), a Python library for programming and executing data-oriented workflows in parallel, addresses these problems. Developers simply annotate a Python script with Parsl directives wrapping either Python functions or calls to external applications. Parsl manages the execution of the script on clusters, clouds, grids, and other resources; orchestrates required data movement; and manages the execution of Python functions and external applications in parallel. The Parsl library can be easily integrated into Python-based gateways, allowing for simple management and scaling of workflows.
KW - Parallel scripting
KW - Parsl
KW - Python
KW - Scientific Workflows
UR - http://www.scopus.com/inward/record.url?scp=85065527336&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85065527336&partnerID=8YFLogxK
M3 - Conference article
AN - SCOPUS:85065527336
VL - 2357
JO - CEUR Workshop Proceedings
JF - CEUR Workshop Proceedings
SN - 1613-0073
ER -