Near-Lossless MPI Tracing and Proxy Application Autogeneration

Chen Wang, Yanfei Guo, Pavan Balaji, Marc Snir

Research output: Contribution to journalArticlepeer-review


Traces of MPI communications are used by many performance analysis and visualization tools. Storing exhaustive traces of large-scale MPI applications is infeasible, however, because of their large volume. Aggregated or lossy MPI traces are smaller but provide much less information. In this paper we present Pilgrim, a near-lossless MPI tracing tool that, by using sophisticated compression techniques, generates small trace files at large scales and incurs only moderate overheads. We perform comprehensive studies of various compression techniques used for storing timestamps associated with each call. This timing information is essential for analysis purposes such as skews study. To demonstrate the usefulness of the detailed information stored by Pilgrim, we present a proxy application generator that can generate proxy apps that preserve original communication patterns from the Pilgrim traces.

Original languageEnglish (US)
Pages (from-to)123-140
Number of pages18
JournalIEEE Transactions on Parallel and Distributed Systems
Issue number1
StatePublished - Jan 1 2023
Externally publishedYes


  • Codes
  • Communication tracing
  • Generators
  • Grammar
  • MPI tracing
  • Production
  • Proxy application generation
  • Standards
  • Symbols
  • Synchronization
  • proxy application generation

ASJC Scopus subject areas

  • Signal Processing
  • Hardware and Architecture
  • Computational Theory and Mathematics


Dive into the research topics of 'Near-Lossless MPI Tracing and Proxy Application Autogeneration'. Together they form a unique fingerprint.

Cite this