Cyber-infrastructure to support science and data management for the dark energy survey

C. Ngeow; J. J. Mohr; T. Alam; W. A. Barkhouse; C. Beldica; D. Cai; Gregory Daues; R. Plante; J. Annis; H. Lin; D. Tucker; R. C. Smith

doi:10.1117/12.671017

Cyber-infrastructure to support science and data management for the dark energy survey

C. Ngeow, J. J. Mohr, T. Alam, W. A. Barkhouse, C. Beldica, D. Cai, Gregory Daues, R. Plante, J. Annis, H. Lin, D. Tucker, R. C. Smith

National Center for Supercomputing Applications (NCSA)

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg² in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect mapping experiment, (2) a cosmic shear measurement over 5000 deg², (3) a galaxy angular clustering measurement within redshift shells to redshift=1.35, and (4) distance measurements to 1900 supernovae la. The DES will produce 200 TB of raw data in four bands, These data will be processed into science ready images and catalogs and co-added into deeper, higher quality images and catalogs. In total, the DES dataset will exceed 1 PB, including a 100 TB catalog database that will serve as a key science analysis tool for the astronomy/cosmology community. The data rate, volume, and duration of the survey require a new type of data management (DM) system that (1) offers a high degree of automation and robustness and (2) leverages the existing high performance computing infrastructure to meet the project's DM targets. The DES DM system consists of (1) a gridenabled, flexible and scalable middleware developed at NCSA for the broader scientific community, (2) astronomy modules that build upon community software, and (3) a DES archive to support automated processing and to serve DES catalogs and images to the collaboration and the public. In the recent DES Data Challenge 1 we deployed and tested the first version of the DES DM system, successfully reducing 700 GB of raw simulated images into 5 TB of reduced data products and cataloguing 50 million objects with calibrated astrometry and photometry.

Original language	English (US)
Title of host publication	Observatory Operations
Subtitle of host publication	Strategies, Processes, and Systems
DOIs	https://doi.org/10.1117/12.671017
State	Published - 2006
Event	Observatory Operations: Strategies, Processes, and Systems - Orlando, FL, United States Duration: May 25 2006 → May 27 2006

Publication series

Name	Proceedings of SPIE - The International Society for Optical Engineering
Volume	6270
ISSN (Print)	0277-786X

Other

Other	Observatory Operations: Strategies, Processes, and Systems
Country/Territory	United States
City	Orlando, FL
Period	5/25/06 → 5/27/06

Keywords

Catalog and archive
Cosmology
Dark energy
Data management
Database
Grid computing
Survey
TeraGrid middleware

ASJC Scopus subject areas

Electronic, Optical and Magnetic Materials
Condensed Matter Physics
Computer Science Applications
Applied Mathematics
Electrical and Electronic Engineering

Online availability

10.1117/12.671017

Library availability

Discover UIUC Full Text

Cite this

Ngeow, C., Mohr, J. J., Alam, T., Barkhouse, W. A., Beldica, C., Cai, D., Daues, G., Plante, R., Annis, J., Lin, H., Tucker, D., & Smith, R. C. (2006). Cyber-infrastructure to support science and data management for the dark energy survey. In Observatory Operations: Strategies, Processes, and Systems Article 627023 (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 6270). https://doi.org/10.1117/12.671017

Cyber-infrastructure to support science and data management for the dark energy survey. / Ngeow, C.; Mohr, J. J.; Alam, T. et al.
Observatory Operations: Strategies, Processes, and Systems. 2006. 627023 (Proceedings of SPIE - The International Society for Optical Engineering; Vol. 6270).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Ngeow, C, Mohr, JJ, Alam, T, Barkhouse, WA, Beldica, C, Cai, D, Daues, G, Plante, R, Annis, J, Lin, H, Tucker, D & Smith, RC 2006, Cyber-infrastructure to support science and data management for the dark energy survey. in Observatory Operations: Strategies, Processes, and Systems., 627023, Proceedings of SPIE - The International Society for Optical Engineering, vol. 6270, Observatory Operations: Strategies, Processes, and Systems, Orlando, FL, United States, 5/25/06. https://doi.org/10.1117/12.671017

@inproceedings{8e2245832e084ee98d226bd1cded008d,

title = "Cyber-infrastructure to support science and data management for the dark energy survey",

abstract = "The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg2 in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect mapping experiment, (2) a cosmic shear measurement over 5000 deg2, (3) a galaxy angular clustering measurement within redshift shells to redshift=1.35, and (4) distance measurements to 1900 supernovae la. The DES will produce 200 TB of raw data in four bands, These data will be processed into science ready images and catalogs and co-added into deeper, higher quality images and catalogs. In total, the DES dataset will exceed 1 PB, including a 100 TB catalog database that will serve as a key science analysis tool for the astronomy/cosmology community. The data rate, volume, and duration of the survey require a new type of data management (DM) system that (1) offers a high degree of automation and robustness and (2) leverages the existing high performance computing infrastructure to meet the project's DM targets. The DES DM system consists of (1) a gridenabled, flexible and scalable middleware developed at NCSA for the broader scientific community, (2) astronomy modules that build upon community software, and (3) a DES archive to support automated processing and to serve DES catalogs and images to the collaboration and the public. In the recent DES Data Challenge 1 we deployed and tested the first version of the DES DM system, successfully reducing 700 GB of raw simulated images into 5 TB of reduced data products and cataloguing 50 million objects with calibrated astrometry and photometry.",

keywords = "Catalog and archive, Cosmology, Dark energy, Data management, Database, Grid computing, Survey, TeraGrid middleware",

author = "C. Ngeow and Mohr, {J. J.} and T. Alam and Barkhouse, {W. A.} and C. Beldica and D. Cai and Gregory Daues and R. Plante and J. Annis and H. Lin and D. Tucker and Smith, {R. C.}",

year = "2006",

doi = "10.1117/12.671017",

language = "English (US)",

isbn = "0819463353",

series = "Proceedings of SPIE - The International Society for Optical Engineering",

booktitle = "Observatory Operations",

note = "Observatory Operations: Strategies, Processes, and Systems ; Conference date: 25-05-2006 Through 27-05-2006",

}

TY - GEN

T1 - Cyber-infrastructure to support science and data management for the dark energy survey

AU - Ngeow, C.

AU - Mohr, J. J.

AU - Alam, T.

AU - Barkhouse, W. A.

AU - Beldica, C.

AU - Cai, D.

AU - Daues, Gregory

AU - Plante, R.

AU - Annis, J.

AU - Lin, H.

AU - Tucker, D.

AU - Smith, R. C.

PY - 2006

Y1 - 2006

N2 - The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg2 in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect mapping experiment, (2) a cosmic shear measurement over 5000 deg2, (3) a galaxy angular clustering measurement within redshift shells to redshift=1.35, and (4) distance measurements to 1900 supernovae la. The DES will produce 200 TB of raw data in four bands, These data will be processed into science ready images and catalogs and co-added into deeper, higher quality images and catalogs. In total, the DES dataset will exceed 1 PB, including a 100 TB catalog database that will serve as a key science analysis tool for the astronomy/cosmology community. The data rate, volume, and duration of the survey require a new type of data management (DM) system that (1) offers a high degree of automation and robustness and (2) leverages the existing high performance computing infrastructure to meet the project's DM targets. The DES DM system consists of (1) a gridenabled, flexible and scalable middleware developed at NCSA for the broader scientific community, (2) astronomy modules that build upon community software, and (3) a DES archive to support automated processing and to serve DES catalogs and images to the collaboration and the public. In the recent DES Data Challenge 1 we deployed and tested the first version of the DES DM system, successfully reducing 700 GB of raw simulated images into 5 TB of reduced data products and cataloguing 50 million objects with calibrated astrometry and photometry.

AB - The Dark Energy Survey (DES; operations 2009-2015) will address the nature of dark energy using four independent and complementary techniques: (1) a galaxy cluster survey over 4000 deg2 in collaboration with the South Pole Telescope Sunyaev-Zel'dovich effect mapping experiment, (2) a cosmic shear measurement over 5000 deg2, (3) a galaxy angular clustering measurement within redshift shells to redshift=1.35, and (4) distance measurements to 1900 supernovae la. The DES will produce 200 TB of raw data in four bands, These data will be processed into science ready images and catalogs and co-added into deeper, higher quality images and catalogs. In total, the DES dataset will exceed 1 PB, including a 100 TB catalog database that will serve as a key science analysis tool for the astronomy/cosmology community. The data rate, volume, and duration of the survey require a new type of data management (DM) system that (1) offers a high degree of automation and robustness and (2) leverages the existing high performance computing infrastructure to meet the project's DM targets. The DES DM system consists of (1) a gridenabled, flexible and scalable middleware developed at NCSA for the broader scientific community, (2) astronomy modules that build upon community software, and (3) a DES archive to support automated processing and to serve DES catalogs and images to the collaboration and the public. In the recent DES Data Challenge 1 we deployed and tested the first version of the DES DM system, successfully reducing 700 GB of raw simulated images into 5 TB of reduced data products and cataloguing 50 million objects with calibrated astrometry and photometry.

KW - Catalog and archive

KW - Cosmology

KW - Dark energy

KW - Data management

KW - Database

KW - Grid computing

KW - Survey

KW - TeraGrid middleware

UR - http://www.scopus.com/inward/record.url?scp=33749315906&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=33749315906&partnerID=8YFLogxK

U2 - 10.1117/12.671017

DO - 10.1117/12.671017

M3 - Conference contribution

AN - SCOPUS:33749315906

SN - 0819463353

SN - 9780819463357

T3 - Proceedings of SPIE - The International Society for Optical Engineering

BT - Observatory Operations

T2 - Observatory Operations: Strategies, Processes, and Systems

Y2 - 25 May 2006 through 27 May 2006

ER -

Cyber-infrastructure to support science and data management for the dark energy survey

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Online availability

Library availability

Related links

Fingerprint

Cite this