Node-Aware Improvements to Allreduce

Amanda Bienz, Luke Olson, William Gropp

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The MPI-Allreduce collective operation is a core kernel of many parallel codebases, particularly for reductions over a single value per process. The commonly used allreduce recursive-doubling algorithm obtains the lower bound message count, yielding optimality for small reduction sizes based on node-agnostic performance models. However, this algorithm yields duplicate messages between sets of nodes. Node-aware optimizations in MPICH remove duplicate messages through use of a single master process per node, yielding a large number of inactive processes at each inter-node step. In this paper, we present an algorithm that uses the multiple processes available per node to reduce the maximum number of inter-node messages communicated by a single process, improving the performance of allreduce operations, particularly for small message sizes.

Original languageEnglish (US)
Title of host publicationProceedings of ExaMPI 2019
Subtitle of host publicationWorkshop on Exascale MPI - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages19-28
Number of pages10
ISBN (Electronic)9781728160092
DOIs
StatePublished - Nov 2019
Event2019 IEEE/ACM Workshop on Exascale MPI, ExaMPI 2019 - Denver, United States
Duration: Nov 17 2019 → …

Publication series

NameProceedings of ExaMPI 2019: Workshop on Exascale MPI - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference2019 IEEE/ACM Workshop on Exascale MPI, ExaMPI 2019
CountryUnited States
CityDenver
Period11/17/19 → …

ASJC Scopus subject areas

  • Safety, Risk, Reliability and Quality
  • Computer Networks and Communications
  • Hardware and Architecture

Fingerprint Dive into the research topics of 'Node-Aware Improvements to Allreduce'. Together they form a unique fingerprint.

  • Cite this

    Bienz, A., Olson, L., & Gropp, W. (2019). Node-Aware Improvements to Allreduce. In Proceedings of ExaMPI 2019: Workshop on Exascale MPI - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 19-28). [8955452] (Proceedings of ExaMPI 2019: Workshop on Exascale MPI - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ExaMPI49596.2019.00008