A massively scalable distributed multigrid framework for nonlinear marine hydrodynamics

Stefan Lemvig Glimberg, Allan Peter Engsig-Karup, Luke Olson

Research output: Contribution to journalArticle

Abstract

The focus of this article is on the parallel scalability of a distributed multigrid framework, known as the DTU Compute GPUlab Library, for execution on graphics processing unit (GPU)-accelerated supercomputers. We demonstrate near-ideal weak scalability for a high-order fully nonlinear potential flow (FNPF) time domain model on the Oak Ridge Titan supercomputer, which is equipped with a large number of many-core CPU-GPU nodes. The high-order finite difference scheme for the solver is implemented to expose data locality and scalability, and the linear Laplace solver is based on an iterative multilevel preconditioned defect correction method designed for high-throughput processing and massive parallelism. In this work, the FNPF discretization is based on a multi-block discretization that allows for large-scale simulations. In this setup, each grid block is based on a logically structured mesh with support for curvilinear representation of horizontal block boundaries to allow for an accurate representation of geometric features such as surface-piercing bottom-mounted structures—for example, mono-pile foundations as demonstrated. Unprecedented performance and scalability results are presented for a system of equations that is historically known as being too expensive to solve in practical applications. A novel feature of the potential flow model is demonstrated, being that a modest number of multigrid restrictions is sufficient for fast convergence, improving overall parallel scalability as the coarse grid problem diminishes. In the numerical benchmarks presented, we demonstrate using 8192 modern Nvidia GPUs enabling large-scale and high-resolution nonlinear marine hydrodynamics applications.

Original languageEnglish (US)
Pages (from-to)855-868
Number of pages14
JournalInternational Journal of High Performance Computing Applications
Volume33
Issue number5
DOIs
StatePublished - Sep 1 2019

Fingerprint

Scalability
Hydrodynamics
Potential Flow
Potential flow
Supercomputers
Fully Nonlinear
Supercomputer
Graphics Processing Unit
Discretization
Higher Order
Grid
Defect Correction
Data Locality
Multiblock
Pile foundations
Piercing
Many-core
Domain Model
Flow Time
Ridge

Keywords

  • High-performance computing
  • Laplace problem
  • domain decomposition
  • free surface water waves
  • geometric multigrid
  • heterogeneous computing
  • multi-GPU
  • multi-block solver

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Hardware and Architecture

Cite this

A massively scalable distributed multigrid framework for nonlinear marine hydrodynamics. / Glimberg, Stefan Lemvig; Engsig-Karup, Allan Peter; Olson, Luke.

In: International Journal of High Performance Computing Applications, Vol. 33, No. 5, 01.09.2019, p. 855-868.

Research output: Contribution to journalArticle

@article{021be083572b42a5abff6ac8eeaddc29,
title = "A massively scalable distributed multigrid framework for nonlinear marine hydrodynamics",
abstract = "The focus of this article is on the parallel scalability of a distributed multigrid framework, known as the DTU Compute GPUlab Library, for execution on graphics processing unit (GPU)-accelerated supercomputers. We demonstrate near-ideal weak scalability for a high-order fully nonlinear potential flow (FNPF) time domain model on the Oak Ridge Titan supercomputer, which is equipped with a large number of many-core CPU-GPU nodes. The high-order finite difference scheme for the solver is implemented to expose data locality and scalability, and the linear Laplace solver is based on an iterative multilevel preconditioned defect correction method designed for high-throughput processing and massive parallelism. In this work, the FNPF discretization is based on a multi-block discretization that allows for large-scale simulations. In this setup, each grid block is based on a logically structured mesh with support for curvilinear representation of horizontal block boundaries to allow for an accurate representation of geometric features such as surface-piercing bottom-mounted structures—for example, mono-pile foundations as demonstrated. Unprecedented performance and scalability results are presented for a system of equations that is historically known as being too expensive to solve in practical applications. A novel feature of the potential flow model is demonstrated, being that a modest number of multigrid restrictions is sufficient for fast convergence, improving overall parallel scalability as the coarse grid problem diminishes. In the numerical benchmarks presented, we demonstrate using 8192 modern Nvidia GPUs enabling large-scale and high-resolution nonlinear marine hydrodynamics applications.",
keywords = "High-performance computing, Laplace problem, domain decomposition, free surface water waves, geometric multigrid, heterogeneous computing, multi-GPU, multi-block solver",
author = "Glimberg, {Stefan Lemvig} and Engsig-Karup, {Allan Peter} and Luke Olson",
year = "2019",
month = "9",
day = "1",
doi = "10.1177/1094342019826662",
language = "English (US)",
volume = "33",
pages = "855--868",
journal = "International Journal of High Performance Computing Applications",
issn = "1094-3420",
publisher = "SAGE Publications Inc.",
number = "5",

}

TY - JOUR

T1 - A massively scalable distributed multigrid framework for nonlinear marine hydrodynamics

AU - Glimberg, Stefan Lemvig

AU - Engsig-Karup, Allan Peter

AU - Olson, Luke

PY - 2019/9/1

Y1 - 2019/9/1

N2 - The focus of this article is on the parallel scalability of a distributed multigrid framework, known as the DTU Compute GPUlab Library, for execution on graphics processing unit (GPU)-accelerated supercomputers. We demonstrate near-ideal weak scalability for a high-order fully nonlinear potential flow (FNPF) time domain model on the Oak Ridge Titan supercomputer, which is equipped with a large number of many-core CPU-GPU nodes. The high-order finite difference scheme for the solver is implemented to expose data locality and scalability, and the linear Laplace solver is based on an iterative multilevel preconditioned defect correction method designed for high-throughput processing and massive parallelism. In this work, the FNPF discretization is based on a multi-block discretization that allows for large-scale simulations. In this setup, each grid block is based on a logically structured mesh with support for curvilinear representation of horizontal block boundaries to allow for an accurate representation of geometric features such as surface-piercing bottom-mounted structures—for example, mono-pile foundations as demonstrated. Unprecedented performance and scalability results are presented for a system of equations that is historically known as being too expensive to solve in practical applications. A novel feature of the potential flow model is demonstrated, being that a modest number of multigrid restrictions is sufficient for fast convergence, improving overall parallel scalability as the coarse grid problem diminishes. In the numerical benchmarks presented, we demonstrate using 8192 modern Nvidia GPUs enabling large-scale and high-resolution nonlinear marine hydrodynamics applications.

AB - The focus of this article is on the parallel scalability of a distributed multigrid framework, known as the DTU Compute GPUlab Library, for execution on graphics processing unit (GPU)-accelerated supercomputers. We demonstrate near-ideal weak scalability for a high-order fully nonlinear potential flow (FNPF) time domain model on the Oak Ridge Titan supercomputer, which is equipped with a large number of many-core CPU-GPU nodes. The high-order finite difference scheme for the solver is implemented to expose data locality and scalability, and the linear Laplace solver is based on an iterative multilevel preconditioned defect correction method designed for high-throughput processing and massive parallelism. In this work, the FNPF discretization is based on a multi-block discretization that allows for large-scale simulations. In this setup, each grid block is based on a logically structured mesh with support for curvilinear representation of horizontal block boundaries to allow for an accurate representation of geometric features such as surface-piercing bottom-mounted structures—for example, mono-pile foundations as demonstrated. Unprecedented performance and scalability results are presented for a system of equations that is historically known as being too expensive to solve in practical applications. A novel feature of the potential flow model is demonstrated, being that a modest number of multigrid restrictions is sufficient for fast convergence, improving overall parallel scalability as the coarse grid problem diminishes. In the numerical benchmarks presented, we demonstrate using 8192 modern Nvidia GPUs enabling large-scale and high-resolution nonlinear marine hydrodynamics applications.

KW - High-performance computing

KW - Laplace problem

KW - domain decomposition

KW - free surface water waves

KW - geometric multigrid

KW - heterogeneous computing

KW - multi-GPU

KW - multi-block solver

UR - http://www.scopus.com/inward/record.url?scp=85061189782&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85061189782&partnerID=8YFLogxK

U2 - 10.1177/1094342019826662

DO - 10.1177/1094342019826662

M3 - Article

AN - SCOPUS:85061189782

VL - 33

SP - 855

EP - 868

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

SN - 1094-3420

IS - 5

ER -