Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales

Ian Foster; Mark Ainsworth; Bryce Allen; Julie Bessac; Franck Cappello; Jong Youl Choi; Emil Constantinescu; Philip E. Davis; Sheng Di; Wendy Di; Hanqi Guo; Scott Klasky; Kerstin Kleese Van Dam; Tahsin Kurc; Qing Liu; Abid Malik; Kshitij Mehta; Klaus Mueller; Todd Munson; George Ostouchov; Manish Parashar; Tom Peterka; Line Pouchard; Dingwen Tao; Ozan Tugluk; Stefan Wild; Matthew Wolf; Justin M. Wozniak; Wei Xu; Shinjae Yoo

doi:10.1007/978-3-319-64203-1_1

Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales

Ian Foster, Mark Ainsworth, Bryce Allen, Julie Bessac, Franck Cappello, Jong Youl Choi, Emil Constantinescu, Philip E. Davis, Sheng Di, Wendy Di, Hanqi Guo, Scott Klasky, Kerstin Kleese Van Dam, Tahsin Kurc, Qing Liu, Abid Malik, Kshitij Mehta, Klaus Mueller, Todd Munson, George OstouchovManish Parashar, Tom Peterka, Line Pouchard, Dingwen Tao, Ozan Tugluk, Stefan Wild, Matthew Wolf, Justin M. Wozniak, Wei Xu, Shinjae Yoo

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

A growing disparity between supercomputer computation speeds and I/O rates makes it increasingly infeasible for applications to save all results for offline analysis. Instead, applications must analyze and reduce data online so as to output only those results needed to answer target scientific question(s). This change in focus complicates application and experiment design and introduces algorithmic, implementation, and programming model challenges that are unfamiliar to many scientists and that have major implications for the design of various elements of supercomputer systems. We review these challenges and describe methods and tools that we are developing to enable experimental exploration of algorithmic, software, and system design alternatives.

Original language	English (US)
Title of host publication	Euro-Par 2017
Subtitle of host publication	Parallel Processing - 23rd International Conference on Parallel and Distributed Computing, Proceedings
Editors	Francisco F. Rivera, Tomas F. Pena, Jose C. Cabaleiro
Publisher	Springer
Pages	3-19
Number of pages	17
ISBN (Print)	9783319642024
DOIs	https://doi.org/10.1007/978-3-319-64203-1_1
State	Published - 2017
Externally published	Yes
Event	23rd International Conference on Parallel and Distributed Computing, Euro-Par 2017 - Santiago de Compostela, Spain Duration: Aug 28 2017 → Sep 1 2017

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	10417 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	23rd International Conference on Parallel and Distributed Computing, Euro-Par 2017
Country/Territory	Spain
City	Santiago de Compostela
Period	8/28/17 → 9/1/17

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Online availability

10.1007/978-3-319-64203-1_1

Library availability

Discover UIUC Full Text

Cite this

Foster, I., Ainsworth, M., Allen, B., Bessac, J., Cappello, F., Choi, J. Y., Constantinescu, E., Davis, P. E., Di, S., Di, W., Guo, H., Klasky, S., Van Dam, K. K., Kurc, T., Liu, Q., Malik, A., Mehta, K., Mueller, K., Munson, T., ... Yoo, S. (2017). Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales. In F. F. Rivera, T. F. Pena, & J. C. Cabaleiro (Eds.), Euro-Par 2017: Parallel Processing - 23rd International Conference on Parallel and Distributed Computing, Proceedings (pp. 3-19). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10417 LNCS). Springer. https://doi.org/10.1007/978-3-319-64203-1_1

Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales. / Foster, Ian; Ainsworth, Mark; Allen, Bryce et al.
Euro-Par 2017: Parallel Processing - 23rd International Conference on Parallel and Distributed Computing, Proceedings. ed. / Francisco F. Rivera; Tomas F. Pena; Jose C. Cabaleiro. Springer, 2017. p. 3-19 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 10417 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Foster, I, Ainsworth, M, Allen, B, Bessac, J, Cappello, F, Choi, JY, Constantinescu, E, Davis, PE, Di, S, Di, W, Guo, H, Klasky, S, Van Dam, KK, Kurc, T, Liu, Q, Malik, A, Mehta, K, Mueller, K, Munson, T, Ostouchov, G, Parashar, M, Peterka, T, Pouchard, L, Tao, D, Tugluk, O, Wild, S, Wolf, M, Wozniak, JM, Xu, W & Yoo, S 2017, Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales. in FF Rivera, TF Pena & JC Cabaleiro (eds), Euro-Par 2017: Parallel Processing - 23rd International Conference on Parallel and Distributed Computing, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 10417 LNCS, Springer, pp. 3-19, 23rd International Conference on Parallel and Distributed Computing, Euro-Par 2017, Santiago de Compostela, Spain, 8/28/17. https://doi.org/10.1007/978-3-319-64203-1_1

Foster I, Ainsworth M, Allen B, Bessac J, Cappello F, Choi JY et al. Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales. In Rivera FF, Pena TF, Cabaleiro JC, editors, Euro-Par 2017: Parallel Processing - 23rd International Conference on Parallel and Distributed Computing, Proceedings. Springer. 2017. p. 3-19. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-319-64203-1_1

Foster, Ian ; Ainsworth, Mark ; Allen, Bryce et al. / Computing Just What You Need : Online Data Analysis and Reduction at Extreme Scales. Euro-Par 2017: Parallel Processing - 23rd International Conference on Parallel and Distributed Computing, Proceedings. editor / Francisco F. Rivera ; Tomas F. Pena ; Jose C. Cabaleiro. Springer, 2017. pp. 3-19 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{cebba15cf7654fcb9b21c58130c8434c,

title = "Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales",

abstract = "A growing disparity between supercomputer computation speeds and I/O rates makes it increasingly infeasible for applications to save all results for offline analysis. Instead, applications must analyze and reduce data online so as to output only those results needed to answer target scientific question(s). This change in focus complicates application and experiment design and introduces algorithmic, implementation, and programming model challenges that are unfamiliar to many scientists and that have major implications for the design of various elements of supercomputer systems. We review these challenges and describe methods and tools that we are developing to enable experimental exploration of algorithmic, software, and system design alternatives.",

author = "Ian Foster and Mark Ainsworth and Bryce Allen and Julie Bessac and Franck Cappello and Choi, {Jong Youl} and Emil Constantinescu and Davis, {Philip E.} and Sheng Di and Wendy Di and Hanqi Guo and Scott Klasky and {Van Dam}, {Kerstin Kleese} and Tahsin Kurc and Qing Liu and Abid Malik and Kshitij Mehta and Klaus Mueller and Todd Munson and George Ostouchov and Manish Parashar and Tom Peterka and Line Pouchard and Dingwen Tao and Ozan Tugluk and Stefan Wild and Matthew Wolf and Wozniak, {Justin M.} and Wei Xu and Shinjae Yoo",

note = "Funding Information: Acknowledgments. This research was supported in part by the Exascale Computing Project (17-SC-20-SC) of the U.S. Department of Energy (DOE), and by DOE{\textquoteright}s Advanced Scientific Research Office (ASCR) under contract DE-AC02-06CH11357. Publisher Copyright: {\textcopyright} 2017, Springer International Publishing AG.; 23rd International Conference on Parallel and Distributed Computing, Euro-Par 2017 ; Conference date: 28-08-2017 Through 01-09-2017",

year = "2017",

doi = "10.1007/978-3-319-64203-1_1",

language = "English (US)",

isbn = "9783319642024",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "3--19",

editor = "Rivera, {Francisco F.} and Pena, {Tomas F.} and Cabaleiro, {Jose C.}",

booktitle = "Euro-Par 2017",

address = "Germany",

}

TY - GEN

T1 - Computing Just What You Need

T2 - 23rd International Conference on Parallel and Distributed Computing, Euro-Par 2017

AU - Foster, Ian

AU - Ainsworth, Mark

AU - Allen, Bryce

AU - Bessac, Julie

AU - Cappello, Franck

AU - Choi, Jong Youl

AU - Constantinescu, Emil

AU - Davis, Philip E.

AU - Di, Sheng

AU - Di, Wendy

AU - Guo, Hanqi

AU - Klasky, Scott

AU - Van Dam, Kerstin Kleese

AU - Kurc, Tahsin

AU - Liu, Qing

AU - Malik, Abid

AU - Mehta, Kshitij

AU - Mueller, Klaus

AU - Munson, Todd

AU - Ostouchov, George

AU - Parashar, Manish

AU - Peterka, Tom

AU - Pouchard, Line

AU - Tao, Dingwen

AU - Tugluk, Ozan

AU - Wild, Stefan

AU - Wolf, Matthew

AU - Wozniak, Justin M.

AU - Xu, Wei

AU - Yoo, Shinjae

N1 - Funding Information: Acknowledgments. This research was supported in part by the Exascale Computing Project (17-SC-20-SC) of the U.S. Department of Energy (DOE), and by DOE’s Advanced Scientific Research Office (ASCR) under contract DE-AC02-06CH11357. Publisher Copyright: © 2017, Springer International Publishing AG.

PY - 2017

Y1 - 2017

N2 - A growing disparity between supercomputer computation speeds and I/O rates makes it increasingly infeasible for applications to save all results for offline analysis. Instead, applications must analyze and reduce data online so as to output only those results needed to answer target scientific question(s). This change in focus complicates application and experiment design and introduces algorithmic, implementation, and programming model challenges that are unfamiliar to many scientists and that have major implications for the design of various elements of supercomputer systems. We review these challenges and describe methods and tools that we are developing to enable experimental exploration of algorithmic, software, and system design alternatives.

AB - A growing disparity between supercomputer computation speeds and I/O rates makes it increasingly infeasible for applications to save all results for offline analysis. Instead, applications must analyze and reduce data online so as to output only those results needed to answer target scientific question(s). This change in focus complicates application and experiment design and introduces algorithmic, implementation, and programming model challenges that are unfamiliar to many scientists and that have major implications for the design of various elements of supercomputer systems. We review these challenges and describe methods and tools that we are developing to enable experimental exploration of algorithmic, software, and system design alternatives.

UR - http://www.scopus.com/inward/record.url?scp=85028716568&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85028716568&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-64203-1_1

DO - 10.1007/978-3-319-64203-1_1

M3 - Conference contribution

AN - SCOPUS:85028716568

SN - 9783319642024

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 3

EP - 19

BT - Euro-Par 2017

A2 - Rivera, Francisco F.

A2 - Pena, Tomas F.

A2 - Cabaleiro, Jose C.

PB - Springer

Y2 - 28 August 2017 through 1 September 2017

ER -

Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this