The Dark Energy Survey Data Management System

Abstract

The Dark Energy Survey collaboration will study cosmic acceleration with a 5000 deg2 griZY survey in the southern sky over 525 nights from 2011-2016. The DES data management (DESDM) system will be used to process and archive these data and the resulting science ready data products. The DESDM system consists of an integrated archive, a processing framework, an ensemble of astronomy codes and a data access framework. We are developing the DESDM system for operation in the high performance computing (HPC) environments at NCSA and Fermilab. Operating the DESDM system in an HPC environment offers both speed and flexibility. We will employ it for our regular nightly processing needs, and for more compute-intensive tasks such as large scale image coaddition campaigns, extraction of weak lensing shear from the full survey dataset, and massive seasonal reprocessing of the DES data. Data products will be available to the Collaboration and later to the public through a virtual-observatory compatible web portal. Our approach leverages investments in publicly available HPC systems, greatly reducing hardware and maintenance costs to the project, which must deploy and maintain only the storage, database platforms and orchestration and web portal nodes that are specific to DESDM. In Fall 2007, we tested the current DESDM system on both simulated and real survey data. We used Teragrid to process 10 simulated DES nights (3TB of raw data), ingesting and calibrating approximately 250 million objects into the DES Archive database. We also used DESDM to process and calibrate over 50 nights of survey data acquired with the Mosaic2 camera. Comparison to truth tables in the case of the simulated data and internal crosschecks in the case of the real data indicate that astrometric and photometric data quality is excellent.

To be published in the proceedings of the SPIE conference on Astronomical Instrumentation (held in Marseille in June 2008). This preprint is made available with the permission of SPIE. Further information together with preprint containing full quality images is available at http://desweb.cosmology.uiuc.edu/wiki

Cited by in corpus (23)

• Dark Energy Survey Year 1 Results: Cosmological Constraints from Galaxy Clustering and Weak Lensing
• Dark Energy Survey Year 1 Results: Constraints on Extended Cosmological Models from Galaxy Clustering and Weak Lensing
• Galaxy clusters discovered via the Sunyaev-Zel'dovich effect in the first 720 square degrees of the South Pole Telescope survey
• Dissecting Photometric redshift for Active Galactic Nuclei using XMM- and Chandra-COSMOS samples
• An SZ-selected sample of the most massive galaxy clusters in the 2500-square-degree South Pole Telescope survey
• The Blanco Cosmology Survey: Data Acquisition, Processing, Calibration, Quality Diagnostics and Data Release
• The Dark Energy Survey Image Processing Pipeline
• Dark Energy Survey Year 1 Results: Measurement of the Baryon Acoustic Oscillation scale in the distribution of galaxies to redshift 1
• X-ray Properties of the First SZE-selected Galaxy Cluster Sample from the South Pole Telescope
• Redshifts, Sample Purity, and BCG Positions for the Galaxy Cluster Catalog from the first 720 Square Degrees of the South Pole Telescope Survey
• Cosmological Constraints from Multiple Probes in the Dark Energy Survey
• Optical Redshift and Richness Estimates for Galaxy Clusters Selected with the Sunyaev-Zel'dovich Effect from 2008 South Pole Telescope Observations
• Testing coupled dark energy with next-generation large-scale observations
• Galaxy Populations in Massive Galaxy Clusters to z=1.1: Color Distribution, Concentration, Halo Occupation Number and Red Sequence Fraction
• Implementation of robust image artifact removal in SWarp through clipped mean stacking
• A Multiband Study of the Galaxy Populations of the First Four Sunyaev--Zeldovich Effect selected Galaxy Clusters
• DES Y1 results: Splitting growth and geometry to test $\Lambda$CDM
• Pan-STARRS1 variability of XMM-COSMOS AGN. I. Impact on photometric redshifts
• Optical-SZE Scaling Relations for DES Optically Selected Clusters within the SPT-SZ Survey
• CosmoDM and its application to Pan-STARRS data
• DECam integration tests on telescope simulator
• Dark Energy Survey Year 1 Results: Photometric Data Set for Cosmology
• The XMM-BCS galaxy cluster survey: I. The X-ray selected cluster catalog from the initial 6 deg${}^2$