Extreme scale survey simulation with python workflows

Antonio Villarreal, Yadu Babuji, Tom Uram, Daniel S. Katz, Kyle Chard, Katrin Heitmann

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) will soon carry out an unprecedented wide, fast, and deep survey of the sky in multiple optical bands. The data from LSST will open up a new discovery space in astronomy and cosmology, simultaneously providing clues toward addressing burning issues of the day, such as the origin of dark energy and and the nature of dark matter, while at the same time yielding data that will, in turn, pose fresh new questions. To prepare for the imminent arrival of this remarkable data set, it is crucial that the associated scientific communities be able to develop the software needed to analyze it. Computational power now available allows us to generate synthetic data sets that can be used as a realistic training ground for such an effort. This effort raises its own challenges - the need to generate very large simulations of the night sky, scaling up simulation campaigns to large numbers of compute nodes across multiple computing centers with different architectures, and optimizing the complex workload around memory requirements and widely varying wall clock times. We describe here a large-scale workflow that melds together Python code to steer the workflow, Parsl to manage the large-scale distributed execution of workflow components, and containers to carry out the image simulation campaign across multiple sites. Taking advantage of these tools, we developed an extreme-scale computational framework and used it to simulate five years of observations for 300 square degrees of sky area. We describe our experiences and lessons learned in developing this workflow capability, and highlight how the scalability and portability of our approach enabled us to efficiently execute it on up to 4000 compute nodes on two supercomputers.

Original languageEnglish (US)
Title of host publicationProceedings - IEEE 17th International Conference on eScience, eScience 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages9
ISBN (Electronic)9781665403610
StatePublished - Sep 2021
Event17th IEEE International Conference on eScience, eScience 2021 - Virtual, Online, Austria
Duration: Sep 20 2021Sep 23 2021

Publication series

NameProceedings - IEEE 17th International Conference on eScience, eScience 2021


Conference17th IEEE International Conference on eScience, eScience 2021
CityVirtual, Online


  • Astronomy
  • Pipeline
  • Simulation
  • Workflow management

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Artificial Intelligence
  • Computer Networks and Communications
  • Hardware and Architecture
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality


Dive into the research topics of 'Extreme scale survey simulation with python workflows'. Together they form a unique fingerprint.

Cite this