The Presync project: Synchronization automation in the cactus framework

Samuel Cupp, Steve R. Brandt, Roland Haas

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The Einstein Toolkit (ETK) is an open-source software platform for computational simulations in relativistic astrophysics and gravitational physics. The Cactus Framework (Cactus) comprises the core component of the ETK and handles creation of distributed data structures, parallelism, I/O, checkpointing, etc. The current Cactus scheduling system relies on the manual synchronization of ghost zones for grid functions (i.e. distributed matrices). Unfortunately, deciding which subroutines should synchronize which grid functions is a non-trivial problem. Incorrect synchronization can result in over-synchronization (a performance problem) or under-synchronization (an error). This issue also creates a barrier for new users and collaborators. We have developed a new scheduling and synchronization method for Cactus to improve the accessibility and efficiency of the ETK. Synchronization and application of boundary conditions are now handled automatically by Cactus, and performance has been improved by as much as twenty percent. The new approach requires each subroutine to have “read” and “write” declarations for individual grid functions. Before scheduled subroutines run, Cactus checks these declarations and performs any synchronization or boundary condition updates as needed. This method removes the difficulty of deciding where synchronization should take place and removes unneeded synchronizations. The only knowledge required to provide these declarations is what grid functions a given subroutine uses and on which parts of the grid they are read or written. In addition, these changes have reduced the number of synchronizations by approximately five percent in unigrid simulations. This new system is a first step in improving Cactus’ scheduling methods to use more advanced and scalable techniques.

Original languageEnglish (US)
Title of host publicationProceedings of the Practice and Experience in Advanced Research Computing
Subtitle of host publicationRise of the Machines (Learning), PEARC 2019
ISBN (Electronic)9781450372275
StatePublished - Jul 28 2019
Event2019 Conference on Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019 - Chicago, United States
Duration: Jul 28 2019Aug 1 2019

Publication series

NamePervasiveHealth: Pervasive Computing Technologies for Healthcare
ISSN (Print)2153-1633


Conference2019 Conference on Practice and Experience in Advanced Research Computing: Rise of the Machines (Learning), PEARC 2019
Country/TerritoryUnited States


  • ACM proceedings
  • Cactus Framework
  • Einstein Toolkit
  • Frameworks
  • Ghost zones
  • Synchronization

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems
  • Computer Science Applications
  • Health Informatics

Cite this