Blue Waters system and component reliability

Brett Bode; David King; Celso L. Mendes; William T. Kramer; Saurabh Jha; Roger Ford; Justin Davis; Steven Dramstad

doi:10.1002/cpe.7978

Blue Waters system and component reliability

Brett Bode, David King, Celso L. Mendes, William T. Kramer, Saurabh Jha, Roger Ford, Justin Davis, Steven Dramstad

Research output: Contribution to journal › Article › peer-review

Abstract

The Blue Waters system, installed in 2012 at NCSA, has the largest component count of any system Cray has built. Blue Waters includes a mix of dual-socket CPU (XE) and single-socket CPU, single GPU (XK) nodes. The primary storage is provided by Cray's Sonexion/ClusterStor Luster storage system delivering 35 PB (raw) storage at 1 TB/s. The statistical failure rates over time for each component including CPU, DIMM, GPU, disk drive, power supply, blower, etc and their impact on higher level failure rates for individual nodes and the systems as a whole are presented in detail, with a particular emphasis on identifying any increases in rate that might indicate the right-side of the expected bathtub curve has been reached. Strategies employed by NCSA and Cray for minimizing the impact of component failure, such as the preemptive removal of suspect disk drives, are also presented.

Original language	English (US)
Article number	e7978
Journal	Concurrency and Computation: Practice and Experience
Volume	36
Issue number	8
DOIs	https://doi.org/10.1002/cpe.7978
State	Published - Apr 10 2024

Keywords

failure analysis
system management

ASJC Scopus subject areas

Software
Theoretical Computer Science
Computer Science Applications
Computer Networks and Communications
Computational Theory and Mathematics

Online availability

10.1002/cpe.7978

Library availability

Discover UIUC Full Text

Cite this

@article{0d513e0e35b94e5bb2d81b208808ed11,

title = "Blue Waters system and component reliability",

abstract = "The Blue Waters system, installed in 2012 at NCSA, has the largest component count of any system Cray has built. Blue Waters includes a mix of dual-socket CPU (XE) and single-socket CPU, single GPU (XK) nodes. The primary storage is provided by Cray's Sonexion/ClusterStor Luster storage system delivering 35 PB (raw) storage at 1 TB/s. The statistical failure rates over time for each component including CPU, DIMM, GPU, disk drive, power supply, blower, etc and their impact on higher level failure rates for individual nodes and the systems as a whole are presented in detail, with a particular emphasis on identifying any increases in rate that might indicate the right-side of the expected bathtub curve has been reached. Strategies employed by NCSA and Cray for minimizing the impact of component failure, such as the preemptive removal of suspect disk drives, are also presented.",

keywords = "failure analysis, system management",

author = "Brett Bode and David King and Mendes, {Celso L.} and Kramer, {William T.} and Saurabh Jha and Roger Ford and Justin Davis and Steven Dramstad",

note = "The Blue Waters sustained-petascale computing project is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993), the National Geospatial-Intelligence Agency and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. We thank the rest of the Blue Waters staff and the HPE/Cray site team, in particular Mark Dalton, for their efforts in keeping Blue Waters in excellent operational shape for the last 8 years. The Blue Waters sustained‐petascale computing project is supported by the National Science Foundation (awards OCI‐0725070 and ACI‐1238993), the National Geospatial‐Intelligence Agency and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana‐Champaign and its National Center for Supercomputing Applications. We thank the rest of the Blue Waters staff and the HPE/Cray site team, in particular Mark Dalton, for their efforts in keeping Blue Waters in excellent operational shape for the last 8 years.",

year = "2024",

month = apr,

day = "10",

doi = "10.1002/cpe.7978",

language = "English (US)",

volume = "36",

journal = "Concurrency and Computation: Practice and Experience",

issn = "1532-0626",

publisher = "John Wiley & Sons, Ltd.",

number = "8",

}

TY - JOUR

T1 - Blue Waters system and component reliability

AU - Bode, Brett

AU - King, David

AU - Mendes, Celso L.

AU - Kramer, William T.

AU - Jha, Saurabh

AU - Ford, Roger

AU - Davis, Justin

AU - Dramstad, Steven

N1 - The Blue Waters sustained-petascale computing project is supported by the National Science Foundation (awards OCI-0725070 and ACI-1238993), the National Geospatial-Intelligence Agency and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana-Champaign and its National Center for Supercomputing Applications. We thank the rest of the Blue Waters staff and the HPE/Cray site team, in particular Mark Dalton, for their efforts in keeping Blue Waters in excellent operational shape for the last 8 years. The Blue Waters sustained‐petascale computing project is supported by the National Science Foundation (awards OCI‐0725070 and ACI‐1238993), the National Geospatial‐Intelligence Agency and the state of Illinois. Blue Waters is a joint effort of the University of Illinois at Urbana‐Champaign and its National Center for Supercomputing Applications. We thank the rest of the Blue Waters staff and the HPE/Cray site team, in particular Mark Dalton, for their efforts in keeping Blue Waters in excellent operational shape for the last 8 years.

PY - 2024/4/10

Y1 - 2024/4/10

N2 - The Blue Waters system, installed in 2012 at NCSA, has the largest component count of any system Cray has built. Blue Waters includes a mix of dual-socket CPU (XE) and single-socket CPU, single GPU (XK) nodes. The primary storage is provided by Cray's Sonexion/ClusterStor Luster storage system delivering 35 PB (raw) storage at 1 TB/s. The statistical failure rates over time for each component including CPU, DIMM, GPU, disk drive, power supply, blower, etc and their impact on higher level failure rates for individual nodes and the systems as a whole are presented in detail, with a particular emphasis on identifying any increases in rate that might indicate the right-side of the expected bathtub curve has been reached. Strategies employed by NCSA and Cray for minimizing the impact of component failure, such as the preemptive removal of suspect disk drives, are also presented.

AB - The Blue Waters system, installed in 2012 at NCSA, has the largest component count of any system Cray has built. Blue Waters includes a mix of dual-socket CPU (XE) and single-socket CPU, single GPU (XK) nodes. The primary storage is provided by Cray's Sonexion/ClusterStor Luster storage system delivering 35 PB (raw) storage at 1 TB/s. The statistical failure rates over time for each component including CPU, DIMM, GPU, disk drive, power supply, blower, etc and their impact on higher level failure rates for individual nodes and the systems as a whole are presented in detail, with a particular emphasis on identifying any increases in rate that might indicate the right-side of the expected bathtub curve has been reached. Strategies employed by NCSA and Cray for minimizing the impact of component failure, such as the preemptive removal of suspect disk drives, are also presented.

KW - failure analysis

KW - system management

UR - http://www.scopus.com/inward/record.url?scp=85179709016&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85179709016&partnerID=8YFLogxK

U2 - 10.1002/cpe.7978

DO - 10.1002/cpe.7978

M3 - Article

AN - SCOPUS:85179709016

SN - 1532-0626

VL - 36

JO - Concurrency and Computation: Practice and Experience

JF - Concurrency and Computation: Practice and Experience

IS - 8

M1 - e7978

ER -

Blue Waters system and component reliability

Abstract

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this