TY - GEN
T1 - Efficient and Cost-Effective HPC on the Cloud
AU - Bhosale, Aditya
AU - Kale, Laxmikant
AU - Kokkila-Schumacher, Sara
N1 - This work is supported by the IBM-Illinois Discovery Accelerator Institute (IIDAI).
PY - 2025/9/9
Y1 - 2025/9/9
N2 - HPC applications are increasingly utilizing cloud resources due to their cost-effectiveness. Among these resources, spot compute instances present an opportunity to run applications at deep discounts compared to on-demand instances. However, they present unique challenges for tightly-coupled HPC applications due to potential interruptions. Traditional parallel programming models like MPI are not inherently fault-tolerant, and existing methods to handle these interruptions are inefficient and require significant programmer effort. In this paper, we present Charm++ as an alternative solution that natively supports fault tolerance, dynamic load balancing, and resource rescaling. We present a tool to run Charm++ applications with a mix of on-demand and spot instances which can detect and efficiently handle spot interruptions without a shared filesystem. We show that using spot instances can result in up to 60% cost savings for our benchmark application.
AB - HPC applications are increasingly utilizing cloud resources due to their cost-effectiveness. Among these resources, spot compute instances present an opportunity to run applications at deep discounts compared to on-demand instances. However, they present unique challenges for tightly-coupled HPC applications due to potential interruptions. Traditional parallel programming models like MPI are not inherently fault-tolerant, and existing methods to handle these interruptions are inefficient and require significant programmer effort. In this paper, we present Charm++ as an alternative solution that natively supports fault tolerance, dynamic load balancing, and resource rescaling. We present a tool to run Charm++ applications with a mix of on-demand and spot instances which can detect and efficiently handle spot interruptions without a shared filesystem. We show that using spot instances can result in up to 60% cost savings for our benchmark application.
KW - HPC
KW - cost-efficiency
KW - fault-tolerance
KW - spot instances
UR - https://www.scopus.com/pages/publications/105017967557
UR - https://www.scopus.com/pages/publications/105017967557#tab=citedBy
U2 - 10.1145/3731545.3744667
DO - 10.1145/3731545.3744667
M3 - Conference contribution
AN - SCOPUS:105017967557
T3 - HPDC 2025 - Proceedings of the 34th International Symposium on High-Performance Parallel and Distributed Computing
BT - HPDC 2025 - Proceedings of the 34th International Symposium on High-Performance Parallel and Distributed Computing
PB - Association for Computing Machinery
T2 - 34th International Symposium on High-Performance Parallel and Distributed Computing
Y2 - 20 July 2025 through 23 July 2025
ER -