The dark energy survey data management system as a data lntensive science gateway

Kailash Kotwani, James Myers, Bill Baker, Joseph Mohr, Greg Daues, Tony Darnell, Bob Armstrong, Terrence McLaren, Joel Plutchak, Dora Cai, Michelle Gower, Shantanu Desai, Ankit Chandra

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

The Dark Energy survey (DEs) collaboration is science effort to understand cosmic acceleration and cdark energy1 responsible for this phenomenon. Dark Energy survey Data Management (DEsDM) system observational astronomy processingpipeline management system that will be used to: process obtained from a survey with the newDEs field camera (DE covering 5ooo sq degree of southern sky; archive final co-added images;extractcatalogs of celestial object every image anddeliver data products to communitythrough portals and services. DEsDM has been designed as a data intensive science Gateway coupling use of shared computational resources (e.g. Teragrid) wi owned databases and file systems for storage distributed across three continents. DEsDM system over the next six years time will perform over 1omillioncPu-hours(sus) of image processing and serve over 4Petabytesof images and 14 objectsto the international DEs collaboration. when delivered for operations in 2o11, it will be one of, if not the, most scalable and powerful systems for processing telescope images, creating co added deep images, and generating detailed catalogs in existence. The project1s software components consist of a processing framework, an ensemble of astronomy codes, integratedarchive,adata-access framework infrastructure. This paper provides an overview of the scope and highlights, the architectural features developed and planned to be able to supportGateway-style management scaleintensive continuous processing and on-demand user queries for analysis.

Original languageEnglish (US)
Title of host publicationProceedings of the 8th International Workshop on Middleware for Grids, Clouds and e-Science, MGC 2010 - Held at the ACM/IFIP/USENIX 11th International Middleware Conference
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450304535
DOIs
StatePublished - Nov 29 2010
Event8th International Workshop on Middleware for Grids, Clouds and e-Science, MGC 2010 - Bangalore, India
Duration: Nov 29 2010Dec 3 2010

Publication series

NameProceedings of the 8th International Workshop on Middleware for Grids, Clouds and e-Science, MGC 2010 - Held at the ACM/IFIP/USENIX 11th International Middleware Conference

Conference

Conference8th International Workshop on Middleware for Grids, Clouds and e-Science, MGC 2010
CountryIndia
CityBangalore
Period11/29/1012/3/10

    Fingerprint

Keywords

  • Collaborative web applications
  • Large scale systems
  • Observational astronomy
  • Pipeline

ASJC Scopus subject areas

  • Software

Cite this

Kotwani, K., Myers, J., Baker, B., Mohr, J., Daues, G., Darnell, T., Armstrong, B., McLaren, T., Plutchak, J., Cai, D., Gower, M., Desai, S., & Chandra, A. (2010). The dark energy survey data management system as a data lntensive science gateway. In Proceedings of the 8th International Workshop on Middleware for Grids, Clouds and e-Science, MGC 2010 - Held at the ACM/IFIP/USENIX 11th International Middleware Conference [9] (Proceedings of the 8th International Workshop on Middleware for Grids, Clouds and e-Science, MGC 2010 - Held at the ACM/IFIP/USENIX 11th International Middleware Conference). Association for Computing Machinery, Inc. https://doi.org/10.1145/1890799.1890808