TEMPI: An Interposed MPI Library with a Canonical Representation of CUDA-aware Datatypes

Carl Pearson, Kun Wu, I. Hsin Chung, Jinjun Xiong, Wen Mei Hwu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

MPI derived datatypes are an abstraction that simplifies handling of non-contiguous data in MPI applications. These datatypes are recursively constructed at runtime from primitive Named Types defined in the MPI standard. More recently, the development and deployment of CUDA-aware MPI implementations has encouraged the transition of distributed high-performance MPI codes to use GPUs. Such implementations allow MPI functions to directly operate on GPU buffers, easing integration of GPU compute into MPI codes. This work first presents a novel datatype handling strategy for nested strided datatypes, which finds a middle ground between the specialized or generic handling in prior work. This work also shows that the performance characteristics of non-contiguous data handling can be modeled with empirical system measurements, and used to transparently improve MPI_Send/Recv latency. Finally, despite substantial attention to non-contiguous GPU data and CUDA-aware MPI implementations, good performance cannot be taken for granted. This work demonstrates its contributions through an MPI interposer library, TEMPI. TEMPI can be used with existing MPI deployments without system or application changes. Ultimately, the interposed-library model of this work demonstrates MPI_Pack speedup of up to 242000x and MPI_Send speedup of up to 59000x compared to the MPI implementation deployed on a leadership-class supercomputer. This yields speedup of more than 917x in a 3D halo exchange with 3072 processes.

Original languageEnglish (US)
Title of host publicationHPDC 2021 - Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing
PublisherAssociation for Computing Machinery, Inc
Pages95-106
Number of pages12
ISBN (Electronic)9781450382175
DOIs
StatePublished - Jun 21 2021
Externally publishedYes
Event30th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2021 - Virtual, Online, Sweden
Duration: Jun 21 2021Jun 25 2021

Publication series

NameHPDC 2021 - Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

Conference

Conference30th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2021
Country/TerritorySweden
CityVirtual, Online
Period6/21/216/25/21

Keywords

  • cuda
  • derived datatype
  • mpi
  • spectrum mpi
  • summit

ASJC Scopus subject areas

  • Computer Science Applications
  • Software
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'TEMPI: An Interposed MPI Library with a Canonical Representation of CUDA-aware Datatypes'. Together they form a unique fingerprint.

Cite this