Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations

Jing Gong, Stefano Markidis, Erwin Laure, Matthew Otten, Paul Fischer, Misun Min

Research output: Contribution to journalArticlepeer-review


We present a hybrid GPU implementation and performance analysis of Nekbone, which represents one of the core kernels of the incompressible Navier–Stokes solver Nek5000. The implementation is based on OpenACC and CUDA Fortran for local parallelization of the compute-intensive matrix–matrix multiplication part, which significantly minimizes the modification of the existing CPU code while extending the simulation capability of the code to GPU architectures. Our discussion includes the GPU results of OpenACC interoperating with CUDA Fortran and the gather–scatter operations with GPUDirect communication. We demonstrate performance of up to 552 Tflops on 16, 384 GPUs of the OLCF Cray XK7 Titan.

Original languageEnglish (US)
Pages (from-to)4160-4180
Number of pages21
JournalJournal of Supercomputing
Issue number11
StatePublished - Nov 1 2016


  • CUDA Fortran
  • GPUDirect
  • Gather–scatter communication
  • Nekbone/Nek5000
  • OpenACC
  • Spectral element discretization

ASJC Scopus subject areas

  • Software
  • Theoretical Computer Science
  • Information Systems
  • Hardware and Architecture


Dive into the research topics of 'Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations'. Together they form a unique fingerprint.

Cite this