Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray

Tingkai Liu; Marquita Ellis; Carlos Costa; Claudia Misale; Sara Kokkila-Schumacher; Jinwook Jung; Gi Joon Nam; Volodymyr Kindratenko

doi:10.1007/978-3-031-40843-4_16

Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray

Tingkai Liu, Marquita Ellis, Carlos Costa, Claudia Misale, Sara Kokkila-Schumacher, Jinwook Jung, Gi Joon Nam, Volodymyr Kindratenko

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

We have extended the Ray framework to enable automatic scaling of workloads on high-performance computing (HPC) clusters managed by SLURM^© and bursting to Cloud managed by Kubernetes^®. Compared to existing HPC-Cloud convergence solutions, our framework demonstrates advantages in several aspects: users can provide their own Cloud resource, the framework provides the Python-level abstraction that does not require users to interact with job submission systems, and allows a single Python-based parallel workload to be run concurrently across an HPC cluster and a Cloud. Applications in Electronic Design Automation are used to demonstrate the functionality of this solution in scaling the workload on an on-premises HPC system and automatically bursting to a public Cloud when running out of allocated HPC resources. The paper focuses on describing the initial implementation and demonstrating novel functionality of the proposed framework as well as identifying practical considerations and limitations for using Cloud bursting mode. The code of our framework is open-sourced.

Original language	English (US)
Title of host publication	High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers
Editors	Amanda Bienz, Michèle Weiland, Marc Baboulin, Carola Kruse
Publisher	Springer
Pages	207-220
Number of pages	14
ISBN (Print)	9783031408427
DOIs	https://doi.org/10.1007/978-3-031-40843-4_16
State	Published - 2023
Event	38th International Conference on High Performance Computing, ISC High Performance 2023 - Hamburg, Germany Duration: May 21 2023 → May 25 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13999 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	38th International Conference on High Performance Computing, ISC High Performance 2023
Country/Territory	Germany
City	Hamburg
Period	5/21/23 → 5/25/23

Keywords

Cloud bursting
HPC
Kubernetes

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Online availability

10.1007/978-3-031-40843-4_16

Library availability

Discover UIUC Full Text

Cite this

Liu, T., Ellis, M., Costa, C., Misale, C., Kokkila-Schumacher, S., Jung, J., Nam, G. J., & Kindratenko, V. (2023). Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray. In A. Bienz, M. Weiland, M. Baboulin, & C. Kruse (Eds.), High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers (pp. 207-220). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13999 LNCS). Springer. https://doi.org/10.1007/978-3-031-40843-4_16

Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray. / Liu, Tingkai; Ellis, Marquita; Costa, Carlos et al.
High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers. ed. / Amanda Bienz; Michèle Weiland; Marc Baboulin; Carola Kruse. Springer, 2023. p. 207-220 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13999 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Liu, T, Ellis, M, Costa, C, Misale, C, Kokkila-Schumacher, S, Jung, J, Nam, GJ & Kindratenko, V 2023, Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray. in A Bienz, M Weiland, M Baboulin & C Kruse (eds), High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13999 LNCS, Springer, pp. 207-220, 38th International Conference on High Performance Computing, ISC High Performance 2023, Hamburg, Germany, 5/21/23. https://doi.org/10.1007/978-3-031-40843-4_16

Liu T, Ellis M, Costa C, Misale C, Kokkila-Schumacher S, Jung J et al. Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray. In Bienz A, Weiland M, Baboulin M, Kruse C, editors, High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers. Springer. 2023. p. 207-220. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-40843-4_16

Liu, Tingkai ; Ellis, Marquita ; Costa, Carlos et al. / Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray. High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers. editor / Amanda Bienz ; Michèle Weiland ; Marc Baboulin ; Carola Kruse. Springer, 2023. pp. 207-220 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9bbd7286c136408bac6a5909442ee4ec,

title = "Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray",

abstract = "We have extended the Ray framework to enable automatic scaling of workloads on high-performance computing (HPC) clusters managed by SLURM{\textcopyright} and bursting to Cloud managed by Kubernetes{\textregistered}. Compared to existing HPC-Cloud convergence solutions, our framework demonstrates advantages in several aspects: users can provide their own Cloud resource, the framework provides the Python-level abstraction that does not require users to interact with job submission systems, and allows a single Python-based parallel workload to be run concurrently across an HPC cluster and a Cloud. Applications in Electronic Design Automation are used to demonstrate the functionality of this solution in scaling the workload on an on-premises HPC system and automatically bursting to a public Cloud when running out of allocated HPC resources. The paper focuses on describing the initial implementation and demonstrating novel functionality of the proposed framework as well as identifying practical considerations and limitations for using Cloud bursting mode. The code of our framework is open-sourced.",

keywords = "Cloud bursting, HPC, Kubernetes",

author = "Tingkai Liu and Marquita Ellis and Carlos Costa and Claudia Misale and Sara Kokkila-Schumacher and Jinwook Jung and Nam, {Gi Joon} and Volodymyr Kindratenko",

note = "Acknowledgement. This work is supported by the IBM-Illinois Discovery Accelerator Institute. This work utilizes resources supported by the National Science Foundation{\textquoteright}s Major Research Instrumentation program, grant #1725729, as well as the University of Illinois Urbana-Champaign.; 38th International Conference on High Performance Computing, ISC High Performance 2023 ; Conference date: 21-05-2023 Through 25-05-2023",

year = "2023",

doi = "10.1007/978-3-031-40843-4_16",

language = "English (US)",

isbn = "9783031408427",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "207--220",

editor = "Amanda Bienz and Mich{\`e}le Weiland and Marc Baboulin and Carola Kruse",

booktitle = "High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers",

address = "Germany",

}

TY - GEN

T1 - Cloud-Bursting and Autoscaling for Python-Native Scientific Workflows Using Ray

AU - Liu, Tingkai

AU - Ellis, Marquita

AU - Costa, Carlos

AU - Misale, Claudia

AU - Kokkila-Schumacher, Sara

AU - Jung, Jinwook

AU - Nam, Gi Joon

AU - Kindratenko, Volodymyr

N1 - Acknowledgement. This work is supported by the IBM-Illinois Discovery Accelerator Institute. This work utilizes resources supported by the National Science Foundation’s Major Research Instrumentation program, grant #1725729, as well as the University of Illinois Urbana-Champaign.

PY - 2023

Y1 - 2023

N2 - We have extended the Ray framework to enable automatic scaling of workloads on high-performance computing (HPC) clusters managed by SLURM© and bursting to Cloud managed by Kubernetes®. Compared to existing HPC-Cloud convergence solutions, our framework demonstrates advantages in several aspects: users can provide their own Cloud resource, the framework provides the Python-level abstraction that does not require users to interact with job submission systems, and allows a single Python-based parallel workload to be run concurrently across an HPC cluster and a Cloud. Applications in Electronic Design Automation are used to demonstrate the functionality of this solution in scaling the workload on an on-premises HPC system and automatically bursting to a public Cloud when running out of allocated HPC resources. The paper focuses on describing the initial implementation and demonstrating novel functionality of the proposed framework as well as identifying practical considerations and limitations for using Cloud bursting mode. The code of our framework is open-sourced.

AB - We have extended the Ray framework to enable automatic scaling of workloads on high-performance computing (HPC) clusters managed by SLURM© and bursting to Cloud managed by Kubernetes®. Compared to existing HPC-Cloud convergence solutions, our framework demonstrates advantages in several aspects: users can provide their own Cloud resource, the framework provides the Python-level abstraction that does not require users to interact with job submission systems, and allows a single Python-based parallel workload to be run concurrently across an HPC cluster and a Cloud. Applications in Electronic Design Automation are used to demonstrate the functionality of this solution in scaling the workload on an on-premises HPC system and automatically bursting to a public Cloud when running out of allocated HPC resources. The paper focuses on describing the initial implementation and demonstrating novel functionality of the proposed framework as well as identifying practical considerations and limitations for using Cloud bursting mode. The code of our framework is open-sourced.

KW - Cloud bursting

KW - HPC

KW - Kubernetes

UR - http://www.scopus.com/inward/record.url?scp=85171335334&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85171335334&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-40843-4_16

DO - 10.1007/978-3-031-40843-4_16

M3 - Conference contribution

AN - SCOPUS:85171335334

SN - 9783031408427

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 207

EP - 220

BT - High Performance Computing - ISC High Performance 2023 International Workshops, Revised Selected Papers

A2 - Bienz, Amanda

A2 - Weiland, Michèle

A2 - Baboulin, Marc

A2 - Kruse, Carola

PB - Springer

T2 - 38th International Conference on High Performance Computing, ISC High Performance 2023

Y2 - 21 May 2023 through 25 May 2023

ER -