TY - JOUR
T1 - Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory
AU - Karavakis, Edward
AU - Guan, Wen
AU - Yang, Zhaoyu
AU - Maeno, Tadashi
AU - Wenaus, Torre
AU - Adelman-McCarthy, Jennifer
AU - Megino, Fernando Barreiro
AU - De, Kaushik
AU - Dubois, Richard
AU - Gower, Michelle
AU - Jenness, Tim
AU - Klimentov, Alexei
AU - Korchuganova, Tatiana
AU - Kowalik, Mikolaj
AU - Lin, Fa Hui
AU - Nilsson, Paul
AU - Padolski, Sergey
AU - Yang, Wei
AU - Ye, Shuwei
N1 - Publisher Copyright:
© The Authors, published by EDP Sciences.
PY - 2024/5/6
Y1 - 2024/5/6
N2 - The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment's run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory's needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.
AB - The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment's run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory's needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.
UR - http://www.scopus.com/inward/record.url?scp=85212193869&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85212193869&partnerID=8YFLogxK
U2 - 10.1051/epjconf/202429504026
DO - 10.1051/epjconf/202429504026
M3 - Conference article
AN - SCOPUS:85212193869
SN - 2101-6275
VL - 295
JO - EPJ Web of Conferences
JF - EPJ Web of Conferences
M1 - 04026
T2 - 26th International Conference on Computing in High Energy and Nuclear Physics, CHEP 2023
Y2 - 8 May 2023 through 12 May 2023
ER -