Container solutions for HPC systems: A case study of using shifter on blue waters

Maxim Belkin, Hon Wai Leong, Roland Haas, Eliu A. Huerta, Mark Neubauer, Galen Wesley Arnold, David Lesny

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Software container solutions have revolutionized application development approaches by enabling lightweight platform abstractions within the so-called “containers.” Several solutions are being actively developed in attempts to bring the benefits of containers to high-performance computing systems with their stringent security demands on the one hand and fundamental resource sharing requirements on the other. In this paper, we discuss the benefits and short-comings of such solutions when deployed on real HPC systems and applied to production scientific applications. We highlight use cases that are either enabled by or significantly benefit from such solutions. We discuss the efforts by HPC system administrators and support staff to support users of these type of workloads on HPC systems not initially designed with these workloads in mind focusing on NCSA's Blue Waters system.

Original languageEnglish (US)
Title of host publicationPractice and Experience in Advanced Research Computing 2018
Subtitle of host publicationSeamless Creativity, PEARC 2018
PublisherAssociation for Computing Machinery
ISBN (Print)9781450364461
DOIs
StatePublished - Jul 22 2018
Event2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018 - Pittsburgh, United States
Duration: Jul 22 2017Jul 26 2017

Publication series

NameACM International Conference Proceeding Series

Other

Other2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018
CountryUnited States
CityPittsburgh
Period7/22/177/26/17

Fingerprint

Containers
Water

Keywords

  • Data Science
  • Petascale
  • Reproducibility

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications

Cite this

Belkin, M., Leong, H. W., Haas, R., Huerta, E. A., Neubauer, M., Arnold, G. W., & Lesny, D. (2018). Container solutions for HPC systems: A case study of using shifter on blue waters. In Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018 [a43] (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3219104.3219145

Container solutions for HPC systems : A case study of using shifter on blue waters. / Belkin, Maxim; Leong, Hon Wai; Haas, Roland; Huerta, Eliu A.; Neubauer, Mark; Arnold, Galen Wesley; Lesny, David.

Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018. Association for Computing Machinery, 2018. a43 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Belkin, M, Leong, HW, Haas, R, Huerta, EA, Neubauer, M, Arnold, GW & Lesny, D 2018, Container solutions for HPC systems: A case study of using shifter on blue waters. in Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018., a43, ACM International Conference Proceeding Series, Association for Computing Machinery, 2018 Practice and Experience in Advanced Research Computing Conference: Seamless Creativity, PEARC 2018, Pittsburgh, United States, 7/22/17. https://doi.org/10.1145/3219104.3219145
Belkin M, Leong HW, Haas R, Huerta EA, Neubauer M, Arnold GW et al. Container solutions for HPC systems: A case study of using shifter on blue waters. In Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018. Association for Computing Machinery. 2018. a43. (ACM International Conference Proceeding Series). https://doi.org/10.1145/3219104.3219145
Belkin, Maxim ; Leong, Hon Wai ; Haas, Roland ; Huerta, Eliu A. ; Neubauer, Mark ; Arnold, Galen Wesley ; Lesny, David. / Container solutions for HPC systems : A case study of using shifter on blue waters. Practice and Experience in Advanced Research Computing 2018: Seamless Creativity, PEARC 2018. Association for Computing Machinery, 2018. (ACM International Conference Proceeding Series).
@inproceedings{3c07d6d5556446e5b9d1f3dc9d403e64,
title = "Container solutions for HPC systems: A case study of using shifter on blue waters",
abstract = "Software container solutions have revolutionized application development approaches by enabling lightweight platform abstractions within the so-called “containers.” Several solutions are being actively developed in attempts to bring the benefits of containers to high-performance computing systems with their stringent security demands on the one hand and fundamental resource sharing requirements on the other. In this paper, we discuss the benefits and short-comings of such solutions when deployed on real HPC systems and applied to production scientific applications. We highlight use cases that are either enabled by or significantly benefit from such solutions. We discuss the efforts by HPC system administrators and support staff to support users of these type of workloads on HPC systems not initially designed with these workloads in mind focusing on NCSA's Blue Waters system.",
keywords = "Data Science, Petascale, Reproducibility",
author = "Maxim Belkin and Leong, {Hon Wai} and Roland Haas and Huerta, {Eliu A.} and Mark Neubauer and Arnold, {Galen Wesley} and David Lesny",
year = "2018",
month = "7",
day = "22",
doi = "10.1145/3219104.3219145",
language = "English (US)",
isbn = "9781450364461",
series = "ACM International Conference Proceeding Series",
publisher = "Association for Computing Machinery",
booktitle = "Practice and Experience in Advanced Research Computing 2018",

}

TY - GEN

T1 - Container solutions for HPC systems

T2 - A case study of using shifter on blue waters

AU - Belkin, Maxim

AU - Leong, Hon Wai

AU - Haas, Roland

AU - Huerta, Eliu A.

AU - Neubauer, Mark

AU - Arnold, Galen Wesley

AU - Lesny, David

PY - 2018/7/22

Y1 - 2018/7/22

N2 - Software container solutions have revolutionized application development approaches by enabling lightweight platform abstractions within the so-called “containers.” Several solutions are being actively developed in attempts to bring the benefits of containers to high-performance computing systems with their stringent security demands on the one hand and fundamental resource sharing requirements on the other. In this paper, we discuss the benefits and short-comings of such solutions when deployed on real HPC systems and applied to production scientific applications. We highlight use cases that are either enabled by or significantly benefit from such solutions. We discuss the efforts by HPC system administrators and support staff to support users of these type of workloads on HPC systems not initially designed with these workloads in mind focusing on NCSA's Blue Waters system.

AB - Software container solutions have revolutionized application development approaches by enabling lightweight platform abstractions within the so-called “containers.” Several solutions are being actively developed in attempts to bring the benefits of containers to high-performance computing systems with their stringent security demands on the one hand and fundamental resource sharing requirements on the other. In this paper, we discuss the benefits and short-comings of such solutions when deployed on real HPC systems and applied to production scientific applications. We highlight use cases that are either enabled by or significantly benefit from such solutions. We discuss the efforts by HPC system administrators and support staff to support users of these type of workloads on HPC systems not initially designed with these workloads in mind focusing on NCSA's Blue Waters system.

KW - Data Science

KW - Petascale

KW - Reproducibility

UR - http://www.scopus.com/inward/record.url?scp=85051436045&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85051436045&partnerID=8YFLogxK

U2 - 10.1145/3219104.3219145

DO - 10.1145/3219104.3219145

M3 - Conference contribution

AN - SCOPUS:85051436045

SN - 9781450364461

T3 - ACM International Conference Proceeding Series

BT - Practice and Experience in Advanced Research Computing 2018

PB - Association for Computing Machinery

ER -