The Astro-WISE Optical Image Pipeline: Development and Implementation

Abstract

We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower. The system is characterized by integration of archiving, calibration, and post-calibration analysis of data from raw, through intermediate, to final data products. It is a true integration thanks to complete linking of data lineage from the final catalogs back to the raw data. This paper describes the pipeline processing of optical wide-field astronomical data from the WFI (http://www.eso.org/lasilla/instruments/wfi/) and OmegaCAM (http://www.astro-wise.org/~omegacam/) instruments using the Astro-WISE information system (the Astro-WISE Environment or simply AWE). This information system is an environment of hardware resources and humanpower distributed over Europe. AWE is characterized by integration of archiving, data calibration, post-calibration analysis, and archiving of raw, intermediate, and final data products. The true integration enables a complete data processing cycle from the raw data up to the publication of science-ready catalogs. The advantages of this system for very large datasets are in the areas of: survey operations management, quality control, calibration analyses, and massive processing.

Accepted for publication in topical issue of Experimental Astronomy on Astro-WISE information system

References in corpus (1)

• Astro-WISE: Chaining to the Universe

Cited by in corpus (50)

• Automated data reduction workflows for astronomy
• Gravitational Lensing Analysis of the Kilo Degree Survey
• The Kilo-Degree Survey
• The fourth data release of the Kilo-Degree Survey: ugri imaging and nine-band optical-IR photometry over 1000 square degrees
• The third data release of the Kilo-Degree Survey and associated data products
• Dark matter halo properties of GAMA galaxy groups from 100 square degrees of KiDS weak lensing data
• KiDS-1000 catalogue: Weak gravitational lensing shear measurements
• KiDS-1000 Methodology: Modelling and inference for joint weak gravitational lensing and spectroscopic galaxy clustering analysis
• LinKS: Discovering galaxy-scale strong lenses in the Kilo-Degree Survey using Convolutional Neural Networks
• The stellar-to-halo mass relation of GAMA galaxies from 100 square degrees of KiDS weak lensing data
• The SAMI Galaxy Survey: mass as the driver of the kinematic morphology - density relation in clusters
• The SAMI Galaxy Survey: The cluster redshift survey, target selection and cluster properties
• First discoveries of z~6 quasars with the Kilo Degree Survey and VISTA Kilo-Degree Infrared Galaxy survey
• The Fornax Deep Survey (FDS) with VST. VI. Optical properties of the dwarf galaxies in the Fornax cluster
• AMICO galaxy clusters in KiDS-DR3: weak-lensing mass calibration
• The SAMI Galaxy Survey: Quenching of star formation in clusters I. Transition galaxies
• The evolution of ultra-diffuse galaxies in nearby galaxy clusters from the Kapteyn IAC WEAVE INT Clusters Survey ($\text{KIWICS}$)
• The Weak Lensing Radial Acceleration Relation: Constraining Modified Gravity and Cold Dark Matter theories with KiDS-1000
• Studying galaxy troughs and ridges using Weak Gravitational Lensing with the Kilo-Degree Survey
• The masses of satellites in GAMA galaxy groups from 100 square degrees of KiDS weak lensing data
• Bright galaxy sample in the Kilo-Degree Survey Data Release 4: selection, photometric redshifts, and physical properties
• Machine Learning based photometric redshifts for the KiDS ESO DR2 galaxies
• Reviewing the frequency and central depletion of Ultra-Diffuse Galaxies in galaxy clusters from the KIWICS survey
• The Fornax Deep Survey with VST. IX. The catalog of sources in the FDS area, with an example study for globular clusters and background galaxies
• The fifth data release of the Kilo Degree Survey: Multi-epoch optical/NIR imaging covering wide and legacy-calibration fields
• A cooperative approach among methods for photometric redshifts estimation: an application to KiDS data
• Dependence of GAMA galaxy halo masses on the cosmic web environment from 100 square degrees of KiDS weak lensing data
• Does the virial mass drive the intra-cluster light? The relationship between the ICL and M${}_{vir}$ from VEGAS
• KiDS-Legacy calibration: unifying shear and redshift calibration with the SKiLLS multi-band image simulations
• VEGAS: A VST Early-type GAlaxy Survey.VI. The diffuse light in HCG 86 from the ultra-deep VEGAS images
• VEGAS: VST Early-type GAlaxy Survey. V. IC 1459 group: Mass assembly history in low density environments
• KiDS-SBI: Simulation-based inference analysis of KiDS-1000 cosmic shear
• Spherical Bayesian mass-mapping with uncertainties: full sky observations on the celestial sphere
• Ultra-compact dwarfs beyond the centre of the Fornax galaxy cluster: Hints of UCD formation in low-density environments
• The Astro-WISE datacentric information system
• On the assembly history of massive galaxies. A pilot project with VEGAS deep imaging and M3G integral field spectroscopy
• MHONGOOSE discovery of a gas-rich low-surface brightness galaxy in the Dorado Group
• Unveiling the environment and faint features of the isolated galaxy CIG 96 with deep optical and HI observations
• The AGN fuelling/feedback cycle in nearby radio galaxies - V. The cold atomic gas of NGC 3100 and its group
• YMCA-1: a new remote star cluster of the Milky Way?
• Mining archival data from wide-field astronomical surveys in search of near-Earth objects
• KMHK 1762: Another star cluster in the Large Magellanic Cloud age gap
• Dynamical cluster masses from photometric surveys
• The YMCA (Yes, Magellanic Clouds Again) survey: probing the outer regions of the Magellanic system with VST
• The Data Zoo in Astro-WISE
• The intragroup light in KiDS+GAMA groups: A stacking analysis
• Hybrid-z: Enhancing Kilo-Degree Survey bright galaxy sample photometric redshifts with deep learning
• The nature of the X-ray sources in dwarf galaxies in nearby clusters from the KIWICS
• AMICO galaxy clusters in KiDS-DR3: the impact of estimator statistics on the luminosity-mass scaling relation
• The Fornax Deep Survey (FDS) with the VST: III. Low Surface Brightness (LSB) dwarfs and Ultra Diffuse Galaxies (UDGs) in the center of the Fornax cluster