Big data and extreme-scale computing: Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry

M. Asch, T. Moore, R. Badia, M. Beck, P. Beckman, T. Bidot, F. Bodin, F. Cappello, A. Choudhary, B. de Supinski, E. Deelman, J. Dongarra, A. Dubey, G. Fox, H. Fu, S. Girona, W. Gropp, M. Heroux, Y. Ishikawa, K. KeaheyD. Keyes, W. Kramer, J. F. Lavignon, Y. Lu, S. Matsuoka, B. Mohr, D. Reed, S. Requena, J. Saltz, T. Schulthess, R. Stevens, M. Swany, A. Szalay, W. Tang, G. Varoquaux, J. P. Vilotte, R. Wisniewski, Z. Xu, I. Zacharov

Research output: Contribution to journalReview article

Abstract

Over the past four years, the Big Data and Exascale Computing (BDEC) project organized a series of five international workshops that aimed to explore the ways in which the new forms of data-centric discovery introduced by the ongoing revolution in high-end data analysis (HDA) might be integrated with the established, simulation-centric paradigm of the high-performance computing (HPC) community. Based on those meetings, we argue that the rapid proliferation of digital data generators, the unprecedented growth in the volume and diversity of the data they generate, and the intense evolution of the methods for analyzing and using that data are radically reshaping the landscape of scientific computing. The most critical problems involve the logistics of wide-area, multistage workflows that will move back and forth across the computing continuum, between the multitude of distributed sensors, instruments and other devices at the networks edge, and the centralized resources of commercial clouds and HPC centers. We suggest that the prospects for the future integration of technological infrastructures and research ecosystems need to be considered at three different levels. First, we discuss the convergence of research applications and workflows that establish a research paradigm that combines both HPC and HDA, where ongoing progress is already motivating efforts at the other two levels. Second, we offer an account of some of the problems involved with creating a converged infrastructure for peripheral environments, that is, a shared infrastructure that can be deployed throughout the network in a scalable manner to meet the highly diverse requirements for processing, communication, and buffering/storage of massive data workflows of many different scientific domains. Third, we focus on some opportunities for software ecosystem convergence in big, logically centralized facilities that execute large-scale simulations and models and/or perform large-scale data analytics. We close by offering some conclusions and recommendations for future investment and policy review.

Original languageEnglish (US)
Pages (from-to)435-479
Number of pages45
JournalInternational Journal of High Performance Computing Applications
Volume32
Issue number4
DOIs
StatePublished - Jul 1 2018
Externally publishedYes

Fingerprint

Ecosystem
Ecosystems
Pathway
Extremes
Software
Computing
Natural sciences computing
Work Flow
Infrastructure
High Performance
Logistics
Data analysis
Paradigm
Communication
Sensors
Distributed Sensor
Processing
Scientific Computing
Strategy
Big data

Keywords

  • Big data
  • extreme-scale computing
  • future software
  • high-end data analysis
  • traditional HPC

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Cite this

Big data and extreme-scale computing : Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry. / Asch, M.; Moore, T.; Badia, R.; Beck, M.; Beckman, P.; Bidot, T.; Bodin, F.; Cappello, F.; Choudhary, A.; de Supinski, B.; Deelman, E.; Dongarra, J.; Dubey, A.; Fox, G.; Fu, H.; Girona, S.; Gropp, W.; Heroux, M.; Ishikawa, Y.; Keahey, K.; Keyes, D.; Kramer, W.; Lavignon, J. F.; Lu, Y.; Matsuoka, S.; Mohr, B.; Reed, D.; Requena, S.; Saltz, J.; Schulthess, T.; Stevens, R.; Swany, M.; Szalay, A.; Tang, W.; Varoquaux, G.; Vilotte, J. P.; Wisniewski, R.; Xu, Z.; Zacharov, I.

In: International Journal of High Performance Computing Applications, Vol. 32, No. 4, 01.07.2018, p. 435-479.

Research output: Contribution to journalReview article

Asch, M, Moore, T, Badia, R, Beck, M, Beckman, P, Bidot, T, Bodin, F, Cappello, F, Choudhary, A, de Supinski, B, Deelman, E, Dongarra, J, Dubey, A, Fox, G, Fu, H, Girona, S, Gropp, W, Heroux, M, Ishikawa, Y, Keahey, K, Keyes, D, Kramer, W, Lavignon, JF, Lu, Y, Matsuoka, S, Mohr, B, Reed, D, Requena, S, Saltz, J, Schulthess, T, Stevens, R, Swany, M, Szalay, A, Tang, W, Varoquaux, G, Vilotte, JP, Wisniewski, R, Xu, Z & Zacharov, I 2018, 'Big data and extreme-scale computing: Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry', International Journal of High Performance Computing Applications, vol. 32, no. 4, pp. 435-479. https://doi.org/10.1177/1094342018778123
Asch, M. ; Moore, T. ; Badia, R. ; Beck, M. ; Beckman, P. ; Bidot, T. ; Bodin, F. ; Cappello, F. ; Choudhary, A. ; de Supinski, B. ; Deelman, E. ; Dongarra, J. ; Dubey, A. ; Fox, G. ; Fu, H. ; Girona, S. ; Gropp, W. ; Heroux, M. ; Ishikawa, Y. ; Keahey, K. ; Keyes, D. ; Kramer, W. ; Lavignon, J. F. ; Lu, Y. ; Matsuoka, S. ; Mohr, B. ; Reed, D. ; Requena, S. ; Saltz, J. ; Schulthess, T. ; Stevens, R. ; Swany, M. ; Szalay, A. ; Tang, W. ; Varoquaux, G. ; Vilotte, J. P. ; Wisniewski, R. ; Xu, Z. ; Zacharov, I. / Big data and extreme-scale computing : Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry. In: International Journal of High Performance Computing Applications. 2018 ; Vol. 32, No. 4. pp. 435-479.
@article{88332520121c4e99aac821c2158d3fbb,
title = "Big data and extreme-scale computing: Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry",
abstract = "Over the past four years, the Big Data and Exascale Computing (BDEC) project organized a series of five international workshops that aimed to explore the ways in which the new forms of data-centric discovery introduced by the ongoing revolution in high-end data analysis (HDA) might be integrated with the established, simulation-centric paradigm of the high-performance computing (HPC) community. Based on those meetings, we argue that the rapid proliferation of digital data generators, the unprecedented growth in the volume and diversity of the data they generate, and the intense evolution of the methods for analyzing and using that data are radically reshaping the landscape of scientific computing. The most critical problems involve the logistics of wide-area, multistage workflows that will move back and forth across the computing continuum, between the multitude of distributed sensors, instruments and other devices at the networks edge, and the centralized resources of commercial clouds and HPC centers. We suggest that the prospects for the future integration of technological infrastructures and research ecosystems need to be considered at three different levels. First, we discuss the convergence of research applications and workflows that establish a research paradigm that combines both HPC and HDA, where ongoing progress is already motivating efforts at the other two levels. Second, we offer an account of some of the problems involved with creating a converged infrastructure for peripheral environments, that is, a shared infrastructure that can be deployed throughout the network in a scalable manner to meet the highly diverse requirements for processing, communication, and buffering/storage of massive data workflows of many different scientific domains. Third, we focus on some opportunities for software ecosystem convergence in big, logically centralized facilities that execute large-scale simulations and models and/or perform large-scale data analytics. We close by offering some conclusions and recommendations for future investment and policy review.",
keywords = "Big data, extreme-scale computing, future software, high-end data analysis, traditional HPC",
author = "M. Asch and T. Moore and R. Badia and M. Beck and P. Beckman and T. Bidot and F. Bodin and F. Cappello and A. Choudhary and {de Supinski}, B. and E. Deelman and J. Dongarra and A. Dubey and G. Fox and H. Fu and S. Girona and W. Gropp and M. Heroux and Y. Ishikawa and K. Keahey and D. Keyes and W. Kramer and Lavignon, {J. F.} and Y. Lu and S. Matsuoka and B. Mohr and D. Reed and S. Requena and J. Saltz and T. Schulthess and R. Stevens and M. Swany and A. Szalay and W. Tang and G. Varoquaux and Vilotte, {J. P.} and R. Wisniewski and Z. Xu and I. Zacharov",
year = "2018",
month = "7",
day = "1",
doi = "10.1177/1094342018778123",
language = "English (US)",
volume = "32",
pages = "435--479",
journal = "International Journal of High Performance Computing Applications",
issn = "1094-3420",
publisher = "SAGE Publications Inc.",
number = "4",

}

TY - JOUR

T1 - Big data and extreme-scale computing

T2 - Pathways to Convergence-Toward a shaping strategy for a future software and data ecosystem for scientific inquiry

AU - Asch, M.

AU - Moore, T.

AU - Badia, R.

AU - Beck, M.

AU - Beckman, P.

AU - Bidot, T.

AU - Bodin, F.

AU - Cappello, F.

AU - Choudhary, A.

AU - de Supinski, B.

AU - Deelman, E.

AU - Dongarra, J.

AU - Dubey, A.

AU - Fox, G.

AU - Fu, H.

AU - Girona, S.

AU - Gropp, W.

AU - Heroux, M.

AU - Ishikawa, Y.

AU - Keahey, K.

AU - Keyes, D.

AU - Kramer, W.

AU - Lavignon, J. F.

AU - Lu, Y.

AU - Matsuoka, S.

AU - Mohr, B.

AU - Reed, D.

AU - Requena, S.

AU - Saltz, J.

AU - Schulthess, T.

AU - Stevens, R.

AU - Swany, M.

AU - Szalay, A.

AU - Tang, W.

AU - Varoquaux, G.

AU - Vilotte, J. P.

AU - Wisniewski, R.

AU - Xu, Z.

AU - Zacharov, I.

PY - 2018/7/1

Y1 - 2018/7/1

N2 - Over the past four years, the Big Data and Exascale Computing (BDEC) project organized a series of five international workshops that aimed to explore the ways in which the new forms of data-centric discovery introduced by the ongoing revolution in high-end data analysis (HDA) might be integrated with the established, simulation-centric paradigm of the high-performance computing (HPC) community. Based on those meetings, we argue that the rapid proliferation of digital data generators, the unprecedented growth in the volume and diversity of the data they generate, and the intense evolution of the methods for analyzing and using that data are radically reshaping the landscape of scientific computing. The most critical problems involve the logistics of wide-area, multistage workflows that will move back and forth across the computing continuum, between the multitude of distributed sensors, instruments and other devices at the networks edge, and the centralized resources of commercial clouds and HPC centers. We suggest that the prospects for the future integration of technological infrastructures and research ecosystems need to be considered at three different levels. First, we discuss the convergence of research applications and workflows that establish a research paradigm that combines both HPC and HDA, where ongoing progress is already motivating efforts at the other two levels. Second, we offer an account of some of the problems involved with creating a converged infrastructure for peripheral environments, that is, a shared infrastructure that can be deployed throughout the network in a scalable manner to meet the highly diverse requirements for processing, communication, and buffering/storage of massive data workflows of many different scientific domains. Third, we focus on some opportunities for software ecosystem convergence in big, logically centralized facilities that execute large-scale simulations and models and/or perform large-scale data analytics. We close by offering some conclusions and recommendations for future investment and policy review.

AB - Over the past four years, the Big Data and Exascale Computing (BDEC) project organized a series of five international workshops that aimed to explore the ways in which the new forms of data-centric discovery introduced by the ongoing revolution in high-end data analysis (HDA) might be integrated with the established, simulation-centric paradigm of the high-performance computing (HPC) community. Based on those meetings, we argue that the rapid proliferation of digital data generators, the unprecedented growth in the volume and diversity of the data they generate, and the intense evolution of the methods for analyzing and using that data are radically reshaping the landscape of scientific computing. The most critical problems involve the logistics of wide-area, multistage workflows that will move back and forth across the computing continuum, between the multitude of distributed sensors, instruments and other devices at the networks edge, and the centralized resources of commercial clouds and HPC centers. We suggest that the prospects for the future integration of technological infrastructures and research ecosystems need to be considered at three different levels. First, we discuss the convergence of research applications and workflows that establish a research paradigm that combines both HPC and HDA, where ongoing progress is already motivating efforts at the other two levels. Second, we offer an account of some of the problems involved with creating a converged infrastructure for peripheral environments, that is, a shared infrastructure that can be deployed throughout the network in a scalable manner to meet the highly diverse requirements for processing, communication, and buffering/storage of massive data workflows of many different scientific domains. Third, we focus on some opportunities for software ecosystem convergence in big, logically centralized facilities that execute large-scale simulations and models and/or perform large-scale data analytics. We close by offering some conclusions and recommendations for future investment and policy review.

KW - Big data

KW - extreme-scale computing

KW - future software

KW - high-end data analysis

KW - traditional HPC

UR - http://www.scopus.com/inward/record.url?scp=85050197185&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050197185&partnerID=8YFLogxK

U2 - 10.1177/1094342018778123

DO - 10.1177/1094342018778123

M3 - Review article

AN - SCOPUS:85050197185

VL - 32

SP - 435

EP - 479

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

SN - 1094-3420

IS - 4

ER -