TY - GEN
T1 - Porting ordinary applications to Blue Gene/Q supercomputers
AU - Maheshwari, Ketan
AU - Wozniak, Justin M.
AU - Armstrong, Timothy G.
AU - Katz, Daniel S.
AU - Binkowski, T. Andrew
AU - Zhong, Xiaoliang
AU - Heinonen, Olle
AU - Karpeyev, Dmitry
AU - Wilde, Michael
N1 - Funding Information:
This work was supported in part by the U.S. Dept. of Energy, Office of Science, Office of Advanced Scientific Computing Research, under Contract DE-AC02-06CH11357. We thank Kevin N. Harms and Raymond M. Loy of ALCF for help with sub-block jobs and application installation. Work by Katz was supported by the National Science Foundation while working at the Foundation. Any opinion, finding, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
Publisher Copyright:
© 2015 IEEE.
PY - 2015/10/22
Y1 - 2015/10/22
N2 - Efficiently porting ordinary applications to Blue Gene/Q supercomputers is a significant challenge. Codes are often originally developed without considering advanced architectures and related tool chains. Science needs frequently lead users to want to run large numbers of relatively small jobs (often called many-task computing, an ensemble, or a workflow), which can conflict with supercomputer configurations. In this paper, we discuss techniques developed to execute ordinary applications over leadership class supercomputers. We use the high-performance Swift parallel scripting framework and build two workflow execution techniques - sub-jobs and main-wrap. The sub-jobs technique, built on top of the IBM Blue Gene/Q resource manager Cobalt's sub-block jobs, lets users submit multiple, independent, repeated smaller jobs within a single larger resource block. The main-wrap technique is a scheme that enables C/C++ programs to be defined as functions that are wrapped by a high-performance Swift wrapper and that are invoked as a Swift script. We discuss the needs, benefits, technicalities, and current limitations of these techniques. We further discuss the real-world science enabled by these techniques and the results obtained.
AB - Efficiently porting ordinary applications to Blue Gene/Q supercomputers is a significant challenge. Codes are often originally developed without considering advanced architectures and related tool chains. Science needs frequently lead users to want to run large numbers of relatively small jobs (often called many-task computing, an ensemble, or a workflow), which can conflict with supercomputer configurations. In this paper, we discuss techniques developed to execute ordinary applications over leadership class supercomputers. We use the high-performance Swift parallel scripting framework and build two workflow execution techniques - sub-jobs and main-wrap. The sub-jobs technique, built on top of the IBM Blue Gene/Q resource manager Cobalt's sub-block jobs, lets users submit multiple, independent, repeated smaller jobs within a single larger resource block. The main-wrap technique is a scheme that enables C/C++ programs to be defined as functions that are wrapped by a high-performance Swift wrapper and that are invoked as a Swift script. We discuss the needs, benefits, technicalities, and current limitations of these techniques. We further discuss the real-world science enabled by these techniques and the results obtained.
KW - BG/Q
KW - Supercomputers
KW - Swift
UR - http://www.scopus.com/inward/record.url?scp=84959053084&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84959053084&partnerID=8YFLogxK
U2 - 10.1109/eScience.2015.8
DO - 10.1109/eScience.2015.8
M3 - Conference contribution
AN - SCOPUS:84959053084
T3 - Proceedings - 11th IEEE International Conference on eScience, eScience 2015
SP - 420
EP - 428
BT - Proceedings - 11th IEEE International Conference on eScience, eScience 2015
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 11th IEEE International Conference on eScience, eScience 2015
Y2 - 31 August 2015 through 4 September 2015
ER -