The Vera C. Rubin Observatory Data Butler and pipeline execution system

Tim Jenness, James F. Bosch, Andrei Salnikov, Nate B. Lust, Nathan M. Pease, Michelle Gower, Mikolaj Kowalik, Gregory P. Dubois-Felsmann, Fritz Mueller, Pim Schellart

Research output: Chapter in Book/Report/Conference proceedingConference contribution


The Rubin Observatory's Data Butler is designed to allow data file location and file formats to be abstracted away from the people writing the science pipeline algorithms. The Butler works in conjunction with the workflow graph builder to allow pipelines to be constructed from the algorithmic tasks. These pipelines can be executed at scale using object stores and multi-node clusters, or on a laptop using a local file system. The Butler and pipeline system are now in daily use during Rubin construction and early operations.

Original languageEnglish (US)
Title of host publicationSoftware and Cyberinfrastructure for Astronomy VII
EditorsJorge Ibsen, Gianluca Chiozzi
ISBN (Electronic)9781510653597
StatePublished - 2022
EventSoftware and Cyberinfrastructure for Astronomy VII 2022 - Montreal, Canada
Duration: Jul 17 2022Jul 21 2022

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


ConferenceSoftware and Cyberinfrastructure for Astronomy VII 2022


  • Data Management
  • Databases
  • Legacy Survey of Space and Time
  • Rubin Observatory

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'The Vera C. Rubin Observatory Data Butler and pipeline execution system'. Together they form a unique fingerprint.

Cite this