Performance Analysis and Optimal Node-aware Communication for Enlarged Conjugate Gradient Methods

Shelby Lockhart, Amanda Bienz, William Gropp, Luke Olson

Research output: Contribution to journalArticlepeer-review


Krylov methods are a key way of solving large sparse linear systems of equations but suffer from poor strong scalability on distributed memory machines. This is due to high synchronization costs from large numbers of collective communication calls alongside a low computational workload. Enlarged Krylov methods address this issue by decreasing the total iterations to convergence, an artifact of splitting the initial residual and resulting in operations on block vectors. In this article, we present a performance study of an enlarged Krylov method, Enlarged Conjugate Gradients (ECG), noting the impact of block vectors on parallel performance at scale. Most notably, we observe the increased overhead of point-to-point communication as a result of denser messages in the sparse matrix-block vector multiplication kernel. Additionally, we present models to analyze expected performance of ECG, as well as motivate design decisions. Most importantly, we introduce a new point-to-point communication approach based on node-aware communication techniques that increases efficiency of the method at scale.

Original languageEnglish (US)
Article number3580003
JournalACM Transactions on Parallel Computing
Issue number1
StatePublished - Mar 29 2023
Externally publishedYes


  • Parallel
  • collectives
  • communication
  • node-aware
  • sparse matrix

ASJC Scopus subject areas

  • Software
  • Modeling and Simulation
  • Hardware and Architecture
  • Computer Science Applications
  • Computational Theory and Mathematics


Dive into the research topics of 'Performance Analysis and Optimal Node-aware Communication for Enlarged Conjugate Gradient Methods'. Together they form a unique fingerprint.

Cite this