Minimizing the usage of hardware counters for collective communication using triggered operations

Nusrat Sharmin Islam, Gengbin Zheng, Sayantan Sur, Akhil Langer, Maria Garzaran

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Triggered operations and counting events or counters are building blocks that can be used by communication libraries, such as MPI, to offload collective operations to the Host Fabric Interface (HFI) or Network Interface Card (NIC). Triggered operations can be used to schedule a network or arithmetic operation to occur in the future, when a trigger counter reaches a specified threshold. On completion of the operation, the value of a completion counter increases by one. With this mechanism, it is possible to create a chain of dependent operations, so that the execution of an operation is triggered when all its dependent operations have completed its execution. Triggered operations rely on hardware counters on the HFI and are a limited resource. Thus, if the number of required counters exceeds the number of hardware counters, a collective needs to stall until a previous collective completes and counters are released. In addition, if the HFI has a counter cache, utilizing a large number of counters can cause cache thrashing and provide poor performance. Therefore, it is important to reduce the number of counters, specially when running on a large supercomputer or when an application uses non-blocking collectives and multiple collectives can run concurrently. In this paper, we propose an algorithm to optimize the number of hardware counters used when offloading collectives with triggered operations. With our algorithm, different operations can share and re-use trigger and completion counters based on the dependences among them and their topological orderings. Our experimental results show that our proposed algorithm significantly reduces the number of counters over a default approach that does not consider the dependences among the operations.

Original languageEnglish (US)
Title of host publicationProceedings of the 26th European MPI Users'' Group Meeting, EuroMPI 2019
EditorsTorsten Hoefler, Jesper Larsson Traff
PublisherAssociation for Computing Machinery
ISBN (Electronic)9781450371759
StatePublished - Sep 11 2019
Externally publishedYes
Event26th European MPI Users'' Group Meeting, EuroMPI 2019 - Zurich, Switzerland
Duration: Sep 11 2019Sep 13 2019

Publication series

NameACM International Conference Proceeding Series


Conference26th European MPI Users'' Group Meeting, EuroMPI 2019

ASJC Scopus subject areas

  • Software
  • Human-Computer Interaction
  • Computer Vision and Pattern Recognition
  • Computer Networks and Communications


Dive into the research topics of 'Minimizing the usage of hardware counters for collective communication using triggered operations'. Together they form a unique fingerprint.

Cite this