TRAM: Optimizing fine-grained communication with topological routing and aggregation of messages

Lukasz Wesolowski, Ramprasad Venkataraman, Abhishek Gupta, Jae Seung Yeom, Keith Bisset, Yanhua Sun, Pritish Jetley, Thomas R. Quinn, Laxmikant V. Kalé

Research output: Contribution to journalConference articlepeer-review


Fine-grained communication in supercomputing applications often limits performance through high communication overhead and poor utilization of network bandwidth. This paper presents Topological Routing and Aggregation Module (TRAM), a library that optimizes fine-grained communication performance by routing and dynamically combining short messages. TRAM collects units of fine-grained communication from the application and combines them into aggregated messages with a common intermediate destination. It routes these messages along a virtual mesh topology mapped onto the physical topology of the network. TRAM improves network bandwidth utilization and reduces communication overhead. It is particularly effective in optimizing patterns with global communication and large message counts, such as all-to-all and many-to-many, as well as sparse, irregular, dynamic or data dependent patterns. We demonstrate how TRAM improves performance through theoretical analysis and experimental verification using benchmarks and scientific applications. We present speedups on petascale systems of 6x for communication benchmarks and up to 4x for applications.

Original languageEnglish (US)
Article number6957230
Pages (from-to)211-220
Number of pages10
JournalProceedings of the International Conference on Parallel Processing
Issue numberNovember
StatePublished - Nov 13 2014
Event43rd International Conference on Parallel Processing, ICPP 2014 - Minneapolis, United States
Duration: Sep 9 2014Sep 12 2014


  • Communication Optimization
  • Interconnection Networks
  • Message Aggregation

ASJC Scopus subject areas

  • Software
  • Mathematics(all)
  • Hardware and Architecture


Dive into the research topics of 'TRAM: Optimizing fine-grained communication with topological routing and aggregation of messages'. Together they form a unique fingerprint.

Cite this