Decoupled I/O for Data-Intensive High Performance Computing

Chao Chen; Yong Chen; Kun Feng; Yanlong Yin; Hassan Eslami; Rajeev Thakur; Xian He Sun; William D. Gropp

doi:10.1109/ICPPW.2014.48

Decoupled I/O for Data-Intensive High Performance Computing

Chao Chen, Yong Chen, Kun Feng, Yanlong Yin, Hassan Eslami, Rajeev Thakur, Xian He Sun, William D. Gropp

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The I/O bottleneck issue has been acknowledged as one of main performance issues of high performance computing (HPC) systems for data-intensive scientific applications, and has attracted intensive studies in recent years. With the enlarging gap between the computing bandwidth and I/O bandwidth in projected next-generation HPC systems, this issue will become even worse. In this paper, we present a novel decoupledI/O to address the fundamental I/O bottleneck issue. The decoupled I/O is a software stack including MPI extensions, compiler improvements, and runtime library support, based one decoupled HPC system architecture. It allows users to treat the computing of data-intensive operations and the traditionalI/O operation as an ensemble and offload them into dedicateddata nodes, which are near to the data source, to reduce the overhead of data movement and improve the I/O bandwidth usage. The decoupled I/O is user-friendly and requires littlechanges in application codes. Experiments were conducted to evaluate the performance of the decoupled I/O, and the results show that it outperforms existing solutions (such as active storage I/O) and provides an attractive I/O solution for data-intensive high performance computing.

Original language	English (US)
Title of host publication	Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	312-320
Number of pages	9
ISBN (Electronic)	9781479956159
DOIs	https://doi.org/10.1109/ICPPW.2014.48
State	Published - May 7 2015
Event	43rd International Conference on Parallel Processing Workshops, ICPPW 2014 - Minneapolis, United States Duration: Sep 9 2014 → Sep 12 2014

Publication series

Name	Proceedings of the International Conference on Parallel Processing Workshops
Volume	2015-May
ISSN (Print)	1530-2016

Other

Other	43rd International Conference on Parallel Processing Workshops, ICPPW 2014
Country	United States
City	Minneapolis
Period	9/9/14 → 9/12/14

Keywords

Decoupled I/O
data-intensive computing
high performance computing
parallel I/O
storage

ASJC Scopus subject areas

Software
Mathematics(all)
Hardware and Architecture

Access to Document

10.1109/ICPPW.2014.48

Fingerprint Dive into the research topics of 'Decoupled I/O for Data-Intensive High Performance Computing'. Together they form a unique fingerprint.

Cite this

Chen, C., Chen, Y., Feng, K., Yin, Y., Eslami, H., Thakur, R., Sun, X. H., & Gropp, W. D. (2015). Decoupled I/O for Data-Intensive High Performance Computing. In Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014 (pp. 312-320). [7103466] (Proceedings of the International Conference on Parallel Processing Workshops; Vol. 2015-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICPPW.2014.48

Decoupled I/O for Data-Intensive High Performance Computing. / Chen, Chao; Chen, Yong; Feng, Kun; Yin, Yanlong; Eslami, Hassan; Thakur, Rajeev; Sun, Xian He; Gropp, William D.

Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014. Institute of Electrical and Electronics Engineers Inc., 2015. p. 312-320 7103466 (Proceedings of the International Conference on Parallel Processing Workshops; Vol. 2015-May).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Chen, C, Chen, Y, Feng, K, Yin, Y, Eslami, H, Thakur, R, Sun, XH & Gropp, WD 2015, Decoupled I/O for Data-Intensive High Performance Computing. in Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014., 7103466, Proceedings of the International Conference on Parallel Processing Workshops, vol. 2015-May, Institute of Electrical and Electronics Engineers Inc., pp. 312-320, 43rd International Conference on Parallel Processing Workshops, ICPPW 2014, Minneapolis, United States, 9/9/14. https://doi.org/10.1109/ICPPW.2014.48

Chen C, Chen Y, Feng K, Yin Y, Eslami H, Thakur R et al. Decoupled I/O for Data-Intensive High Performance Computing. In Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014. Institute of Electrical and Electronics Engineers Inc. 2015. p. 312-320. 7103466. (Proceedings of the International Conference on Parallel Processing Workshops). https://doi.org/10.1109/ICPPW.2014.48

Chen, Chao ; Chen, Yong ; Feng, Kun ; Yin, Yanlong ; Eslami, Hassan ; Thakur, Rajeev ; Sun, Xian He ; Gropp, William D. / Decoupled I/O for Data-Intensive High Performance Computing. Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 312-320 (Proceedings of the International Conference on Parallel Processing Workshops).

@inproceedings{3e206c205a2e4924b5399409aaf0eeb3,

title = "Decoupled I/O for Data-Intensive High Performance Computing",

abstract = "The I/O bottleneck issue has been acknowledged as one of main performance issues of high performance computing (HPC) systems for data-intensive scientific applications, and has attracted intensive studies in recent years. With the enlarging gap between the computing bandwidth and I/O bandwidth in projected next-generation HPC systems, this issue will become even worse. In this paper, we present a novel decoupledI/O to address the fundamental I/O bottleneck issue. The decoupled I/O is a software stack including MPI extensions, compiler improvements, and runtime library support, based one decoupled HPC system architecture. It allows users to treat the computing of data-intensive operations and the traditionalI/O operation as an ensemble and offload them into dedicateddata nodes, which are near to the data source, to reduce the overhead of data movement and improve the I/O bandwidth usage. The decoupled I/O is user-friendly and requires littlechanges in application codes. Experiments were conducted to evaluate the performance of the decoupled I/O, and the results show that it outperforms existing solutions (such as active storage I/O) and provides an attractive I/O solution for data-intensive high performance computing.",

keywords = "Decoupled I/O, data-intensive computing, high performance computing, parallel I/O, storage",

author = "Chao Chen and Yong Chen and Kun Feng and Yanlong Yin and Hassan Eslami and Rajeev Thakur and Sun, {Xian He} and Gropp, {William D.}",

year = "2015",

month = may,

day = "7",

doi = "10.1109/ICPPW.2014.48",

language = "English (US)",

series = "Proceedings of the International Conference on Parallel Processing Workshops",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "312--320",

booktitle = "Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014",

address = "United States",

note = "43rd International Conference on Parallel Processing Workshops, ICPPW 2014 ; Conference date: 09-09-2014 Through 12-09-2014",

}

TY - GEN

T1 - Decoupled I/O for Data-Intensive High Performance Computing

AU - Chen, Chao

AU - Chen, Yong

AU - Feng, Kun

AU - Yin, Yanlong

AU - Eslami, Hassan

AU - Thakur, Rajeev

AU - Sun, Xian He

AU - Gropp, William D.

PY - 2015/5/7

Y1 - 2015/5/7

N2 - The I/O bottleneck issue has been acknowledged as one of main performance issues of high performance computing (HPC) systems for data-intensive scientific applications, and has attracted intensive studies in recent years. With the enlarging gap between the computing bandwidth and I/O bandwidth in projected next-generation HPC systems, this issue will become even worse. In this paper, we present a novel decoupledI/O to address the fundamental I/O bottleneck issue. The decoupled I/O is a software stack including MPI extensions, compiler improvements, and runtime library support, based one decoupled HPC system architecture. It allows users to treat the computing of data-intensive operations and the traditionalI/O operation as an ensemble and offload them into dedicateddata nodes, which are near to the data source, to reduce the overhead of data movement and improve the I/O bandwidth usage. The decoupled I/O is user-friendly and requires littlechanges in application codes. Experiments were conducted to evaluate the performance of the decoupled I/O, and the results show that it outperforms existing solutions (such as active storage I/O) and provides an attractive I/O solution for data-intensive high performance computing.

AB - The I/O bottleneck issue has been acknowledged as one of main performance issues of high performance computing (HPC) systems for data-intensive scientific applications, and has attracted intensive studies in recent years. With the enlarging gap between the computing bandwidth and I/O bandwidth in projected next-generation HPC systems, this issue will become even worse. In this paper, we present a novel decoupledI/O to address the fundamental I/O bottleneck issue. The decoupled I/O is a software stack including MPI extensions, compiler improvements, and runtime library support, based one decoupled HPC system architecture. It allows users to treat the computing of data-intensive operations and the traditionalI/O operation as an ensemble and offload them into dedicateddata nodes, which are near to the data source, to reduce the overhead of data movement and improve the I/O bandwidth usage. The decoupled I/O is user-friendly and requires littlechanges in application codes. Experiments were conducted to evaluate the performance of the decoupled I/O, and the results show that it outperforms existing solutions (such as active storage I/O) and provides an attractive I/O solution for data-intensive high performance computing.

KW - Decoupled I/O

KW - data-intensive computing

KW - high performance computing

KW - parallel I/O

KW - storage

UR - http://www.scopus.com/inward/record.url?scp=84946576832&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84946576832&partnerID=8YFLogxK

U2 - 10.1109/ICPPW.2014.48

DO - 10.1109/ICPPW.2014.48

M3 - Conference contribution

AN - SCOPUS:84946576832

T3 - Proceedings of the International Conference on Parallel Processing Workshops

SP - 312

EP - 320

BT - Proceedings - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 43rd International Conference on Parallel Processing Workshops, ICPPW 2014

Y2 - 9 September 2014 through 12 September 2014

ER -