A massively scalable distributed multigrid framework for nonlinear marine hydrodynamics

Stefan Lemvig Glimberg, Allan Peter Engsig-Karup, Luke N. Olson

Research output: Contribution to journalArticlepeer-review


The focus of this article is on the parallel scalability of a distributed multigrid framework, known as the DTU Compute GPUlab Library, for execution on graphics processing unit (GPU)-accelerated supercomputers. We demonstrate near-ideal weak scalability for a high-order fully nonlinear potential flow (FNPF) time domain model on the Oak Ridge Titan supercomputer, which is equipped with a large number of many-core CPU-GPU nodes. The high-order finite difference scheme for the solver is implemented to expose data locality and scalability, and the linear Laplace solver is based on an iterative multilevel preconditioned defect correction method designed for high-throughput processing and massive parallelism. In this work, the FNPF discretization is based on a multi-block discretization that allows for large-scale simulations. In this setup, each grid block is based on a logically structured mesh with support for curvilinear representation of horizontal block boundaries to allow for an accurate representation of geometric features such as surface-piercing bottom-mounted structures—for example, mono-pile foundations as demonstrated. Unprecedented performance and scalability results are presented for a system of equations that is historically known as being too expensive to solve in practical applications. A novel feature of the potential flow model is demonstrated, being that a modest number of multigrid restrictions is sufficient for fast convergence, improving overall parallel scalability as the coarse grid problem diminishes. In the numerical benchmarks presented, we demonstrate using 8192 modern Nvidia GPUs enabling large-scale and high-resolution nonlinear marine hydrodynamics applications.

Original languageEnglish (US)
Pages (from-to)855-868
Number of pages14
JournalInternational Journal of High Performance Computing Applications
Issue number5
StatePublished - Sep 1 2019


  • High-performance computing
  • Laplace problem
  • domain decomposition
  • free surface water waves
  • geometric multigrid
  • heterogeneous computing
  • multi-GPU
  • multi-block solver

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture


Dive into the research topics of 'A massively scalable distributed multigrid framework for nonlinear marine hydrodynamics'. Together they form a unique fingerprint.

Cite this