Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1

Ken Rafenetti; Abdelhalim Amer; Lena Oden; Charles Archer; Wesley Bland; Hajime Fujita; Yanfei Guo; Tomislav Janjusic; Dmitry Durnov; Michael Blocksome; Min Si; Sangmin Seo; Akhil Langer; Gengbin Zheng; Masamichi Takagi; Paul Cofman; Jithin Jose; Sayantan Sur; Alexander Sannikov; Sergey Oblomov; Michael Chuvelev; Masayuki Hatanaka; Xin Zhao; Paul Fischer; Thilina Rathnayake; Matt Otten; Misun Min; Pavan Balaji

doi:10.1145/3126908.3126963

Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1

Ken Rafenetti, Abdelhalim Amer, Lena Oden, Charles Archer, Wesley Bland, Hajime Fujita, Yanfei Guo, Tomislav Janjusic, Dmitry Durnov, Michael Blocksome, Min Si, Sangmin Seo, Akhil Langer, Gengbin Zheng, Masamichi Takagi, Paul Cofman, Jithin Jose, Sayantan Sur, Alexander Sannikov, Sergey OblomovMichael Chuvelev, Masayuki Hatanaka, Xin Zhao, Paul Fischer, Thilina Rathnayake, Matt Otten, Misun Min, Pavan Balaji

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

This paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 speciication. We irst present a highly optimized implementation of the MPI-3.1 standard in which the communication stackDall the way from the application to the low-level network communication APIDtakes only a few tens of instructions. We carefully study these instructions and analyze the root cause of the overheads based on speciic requirements from the MPI standard that are unavoidable under the current MPI standard. We recommend potential changes to the MPI standard that can minimize these overheads. Our experimental results on a variety of network architectures and applications demonstrate signiicant beneits from our proposed changes.

Original language	English (US)
Title of host publication	Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450351140
DOIs	https://doi.org/10.1145/3126908.3126963
State	Published - Nov 12 2017
Event	International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 - Denver, United States Duration: Nov 12 2017 → Nov 17 2017

Publication series

Name	Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017

Conference

Conference	International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017
Country/Territory	United States
City	Denver
Period	11/12/17 → 11/17/17

ASJC Scopus subject areas

Software
Computer Networks and Communications

Online availability

10.1145/3126908.3126963

Library availability

Discover UIUC Full Text

Cite this

Rafenetti, K., Amer, A., Oden, L., Archer, C., Bland, W., Fujita, H., Guo, Y., Janjusic, T., Durnov, D., Blocksome, M., Si, M., Seo, S., Langer, A., Zheng, G., Takagi, M., Cofman, P., Jose, J., Sur, S., Sannikov, A., ... Balaji, P. (2017). Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 Article 62 (Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017). Association for Computing Machinery. https://doi.org/10.1145/3126908.3126963

Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1. / Rafenetti, Ken; Amer, Abdelhalim; Oden, Lena et al.
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017. Association for Computing Machinery, 2017. 62 (Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Rafenetti, K, Amer, A, Oden, L, Archer, C, Bland, W, Fujita, H, Guo, Y, Janjusic, T, Durnov, D, Blocksome, M, Si, M, Seo, S, Langer, A, Zheng, G, Takagi, M, Cofman, P, Jose, J, Sur, S, Sannikov, A, Oblomov, S, Chuvelev, M, Hatanaka, M, Zhao, X, Fischer, P, Rathnayake, T, Otten, M, Min, M & Balaji, P 2017, Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1. in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017., 62, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Association for Computing Machinery, International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Denver, United States, 11/12/17. https://doi.org/10.1145/3126908.3126963

Rafenetti K, Amer A, Oden L, Archer C, Bland W, Fujita H et al. Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017. Association for Computing Machinery. 2017. 62. (Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017). doi: 10.1145/3126908.3126963

Rafenetti, Ken ; Amer, Abdelhalim ; Oden, Lena et al. / Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017. Association for Computing Machinery, 2017. (Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017).

@inproceedings{99ab9c83a4c74b6fb2439dbb1248a9ab,

title = "Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1",

abstract = "This paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 speciication. We irst present a highly optimized implementation of the MPI-3.1 standard in which the communication stackDall the way from the application to the low-level network communication APIDtakes only a few tens of instructions. We carefully study these instructions and analyze the root cause of the overheads based on speciic requirements from the MPI standard that are unavoidable under the current MPI standard. We recommend potential changes to the MPI standard that can minimize these overheads. Our experimental results on a variety of network architectures and applications demonstrate signiicant beneits from our proposed changes.",

author = "Ken Rafenetti and Abdelhalim Amer and Lena Oden and Charles Archer and Wesley Bland and Hajime Fujita and Yanfei Guo and Tomislav Janjusic and Dmitry Durnov and Michael Blocksome and Min Si and Sangmin Seo and Akhil Langer and Gengbin Zheng and Masamichi Takagi and Paul Cofman and Jithin Jose and Sayantan Sur and Alexander Sannikov and Sergey Oblomov and Michael Chuvelev and Masayuki Hatanaka and Xin Zhao and Paul Fischer and Thilina Rathnayake and Matt Otten and Misun Min and Pavan Balaji",

note = "Publisher Copyright: {\textcopyright} 2017 Copyright held by the owner/author(s).; International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 ; Conference date: 12-11-2017 Through 17-11-2017",

year = "2017",

month = nov,

day = "12",

doi = "10.1145/3126908.3126963",

language = "English (US)",

series = "Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017",

publisher = "Association for Computing Machinery",

booktitle = "Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017",

address = "United States",

}

TY - GEN

T1 - Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1

AU - Rafenetti, Ken

AU - Amer, Abdelhalim

AU - Oden, Lena

AU - Archer, Charles

AU - Bland, Wesley

AU - Fujita, Hajime

AU - Guo, Yanfei

AU - Janjusic, Tomislav

AU - Durnov, Dmitry

AU - Blocksome, Michael

AU - Si, Min

AU - Seo, Sangmin

AU - Langer, Akhil

AU - Zheng, Gengbin

AU - Takagi, Masamichi

AU - Cofman, Paul

AU - Jose, Jithin

AU - Sur, Sayantan

AU - Sannikov, Alexander

AU - Oblomov, Sergey

AU - Chuvelev, Michael

AU - Hatanaka, Masayuki

AU - Zhao, Xin

AU - Fischer, Paul

AU - Rathnayake, Thilina

AU - Otten, Matt

AU - Min, Misun

AU - Balaji, Pavan

PY - 2017/11/12

Y1 - 2017/11/12

N2 - This paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 speciication. We irst present a highly optimized implementation of the MPI-3.1 standard in which the communication stackDall the way from the application to the low-level network communication APIDtakes only a few tens of instructions. We carefully study these instructions and analyze the root cause of the overheads based on speciic requirements from the MPI standard that are unavoidable under the current MPI standard. We recommend potential changes to the MPI standard that can minimize these overheads. Our experimental results on a variety of network architectures and applications demonstrate signiicant beneits from our proposed changes.

AB - This paper provides an in-depth analysis of the software overheads in the MPI performance-critical path and exposes mandatory performance overheads that are unavoidable based on the MPI-3.1 speciication. We irst present a highly optimized implementation of the MPI-3.1 standard in which the communication stackDall the way from the application to the low-level network communication APIDtakes only a few tens of instructions. We carefully study these instructions and analyze the root cause of the overheads based on speciic requirements from the MPI standard that are unavoidable under the current MPI standard. We recommend potential changes to the MPI standard that can minimize these overheads. Our experimental results on a variety of network architectures and applications demonstrate signiicant beneits from our proposed changes.

UR - http://www.scopus.com/inward/record.url?scp=85040178444&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85040178444&partnerID=8YFLogxK

U2 - 10.1145/3126908.3126963

DO - 10.1145/3126908.3126963

M3 - Conference contribution

AN - SCOPUS:85142260100

T3 - Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017

BT - Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017

PB - Association for Computing Machinery

T2 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017

Y2 - 12 November 2017 through 17 November 2017

ER -

Why is MPI so slow? Analyzing the fundamental limits in implementing MPI-3.1

Abstract

Publication series

Conference

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this