Scalable Non-blocking Preconditioned Conjugate Gradient Methods

The preconditioned conjugate gradient method (PCG) is a popular method for solving linear systems at scale. PCG requires frequent blocking allreduce collective operations that can limit performance at scale. We investigate PCG variations designed to reduce communication costs by decreasing the number of allreduces and by overlapping communication with computation using a non-blocking allreduce. These variations include two methods we have developed, non-blocking PCG and 2-step pipelined PCG, and pipelined PCG from Ghysels and Vanroose. Performance modeling for communication and computation costs shows the expected performance of these methods. Weak and strong scaling experiments on up to 128k cores show that scalable PCG methods can outperform standard PCG at scale. We observe that the fastest method varies depending on the work per core, suggesting we need a suite of scalable solvers to obtain the best performance. Experiments with multiple preconditioners and linear systems show the robustness of these methods.