On the realistic validation of photometric redshifts, or why Teddy will never be Happy

Abstract

Two of the main problems encountered in the development and accurate validation of photometric redshift (photo-z) techniques are the lack of spectroscopic coverage in feature space (e.g. colours and magnitudes) and the mismatch between photometric error distributions associated with the spectroscopic and photometric samples. Although these issues are well known, there is currently no standard benchmark allowing a quantitative analysis of their impact on the final photo-z estimation. In this work, we present two galaxy catalogues, Teddy and Happy, built to enable a more demanding and realistic test of photo-z methods. Using photometry from the Sloan Digital Sky Survey and spectroscopy from a collection of sources, we constructed datasets which mimic the biases between the underlying probability distribution of the real spectroscopic and photometric sample. We demonstrate the potential of these catalogues by submitting them to the scrutiny of different photo-z methods, including machine learning (ML) and template fitting approaches. Beyond the expected bad results from most ML algorithms for cases with missing coverage in feature space, we were able to recognize the superiority of global models in the same situation and the general failure across all types of methods when incomplete coverage is convoluted with the presence of photometric errors - a data situation which photo-z methods were not trained to deal with up to now and which must be addressed by future large scale surveys. Our catalogues represent the first controlled environment allowing a straightforward implementation of such tests. The data are publicly available within the COINtoolbox (https://github.com/COINtoolbox/photoz_catalogues).

arXiv comment19 pages, 10 figures. Minor revision accepted by MNRAS on 2017 March 16

References in corpus (65)

• Maps of Dust IR Emission for Use in Estimation of Reddening and CMBR Foregrounds
• Stellar population synthesis at the resolution of 2003
• The Eleventh and Twelfth Data Releases of the Sloan Digital Sky Survey: Final Data from SDSS-III
• The 2dF Galaxy Redshift Survey: Spectra and redshifts
• EAZY: A Fast, Public Photometric Redshift Code
• Accurate photometric redshifts for the CFHT Legacy Survey calibrated using the VIMOS VLT Deep Survey
• K-corrections and filter transformations in the ultraviolet, optical, and near infrared
• COSMOS Photometric Redshifts with 30-bands for 2-deg2
• Bayesian photometric redshift estimation
• Galaxy and Mass Assembly (GAMA): survey diagnostics and core data release
• The 6dF Galaxy Survey: Final Redshift Release (DR3) and Southern Large-Scale Structures
• zCOSMOS: A Large VLT/VIMOS redshift survey covering 0 < z < 3 in the COSMOS field
• Euclid Definition Study Report
• The DEEP2 Galaxy Redshift Survey: Design, Observations, Data Reduction, and Redshifts
• The 6dF Galaxy Survey: Samples, Observational Techniques and the First Data Release
• The WiggleZ Dark Energy Survey: Survey Design and First Data Release
• Science Objectives and Early Results of the DEEP2 Redshift Survey
• ANNz: estimating photometric redshifts using artificial neural networks
• A Modified Magnitude System that Produces Well-Behaved Magnitudes, Colors, and Errors Even for Low Signal-to-Noise Ratio Measurements
• Galaxies in the Hubble Ultra Deep Field: I. Detection, Multiband Photometry, Photometric Redshifts, and Morphology
• A Critical Assessment of Photometric Redshift Methods: A CANDELS Investigation
• The VIMOS Public Extragalactic Redshift Survey (VIPERS). An unprecedented view of galaxies and large-scale structure at 0.5<z<1.2
• The PRIsm MUlti-Object Survey (PRIMUS) I: Survey Overview and Characteristics
• The WiggleZ Dark Energy Survey: Final data release and cosmological results
• PHAT: PHoto-z Accuracy Testing
• The VIMOS Public Extragalactic Survey (VIPERS): First Data Release of 57 204 spectroscopic measurements
• Photometric Redshifts for the SDSS Early Data Release
• Estimating the Redshift Distribution of Faint Galaxy Samples
• Photometric redshifts for the SDSS Data Release 12
• The PRIsm MUlti-object Survey (PRIMUS). II. Data Reduction and Redshift Fitting
• Photometric redshift analysis in the Dark Energy Survey Science Verification data
• The Vimos VLT Deep Survey: Global properties of 20000 galaxies in the I_AB<=22.5 WIDE survey
• Redshift distributions of galaxies in the DES Science Verification shear catalogue and implications for weak lensing
• Galaxy And Mass Assembly (GAMA): autoz spectral redshift measurements, confidence and errors
• Probabilistic Cross-Identification of Astronomical Sources
• The VIRMOS deep imaging survey: I. overview and survey strategy
• Estimating Photometric Redshifts Using Support Vector Machines
• A Comparison of Six Photometric Redshift Methods Applied to 1.5 Million Luminous Red Galaxies
• Photo-z for weak lensing tomography from space: the role of optical and near-IR photometry
• Reconstructing Galaxy Spectral Energy Distributions from Broadband Photometry
• A blind test of photometric redshifts on ground-based data
• Kernel PCA for type Ia supernovae photometric classification
• Towards More Precise Survey Photometry for PanSTARRS and LSST: Measuring Directly the Optical Transmission Spectrum of the Atmosphere
• A catalogue of photometric redshifts for the SDSS-DR9 galaxies
• Hierarchical Bayesian inference of galaxy redshift distributions from photometric surveys
• Machine Learning based photometric redshifts for the KiDS ESO DR2 galaxies
• Hubble parameter reconstruction from a principal component analysis: minimizing the bias
• Photometric redshifts and K-corrections for Sloan Digital Sky Survey Seven Data Release
• GAz: A Genetic Algorithm for Photometric Redshift Estimation
• The Overlooked Potential of Generalized Linear Models in Astronomy - I: Binomial Regression
• A Unified Framework for Photometric Redshifts
• Systematic errors in the measurement of neutrino masses due to baryonic feedback processes: Prospects for stage IV lensing surveys
• Pushing the Limits of Ground-Based Photometric Precision -- Sub-Millimagnitude Time-Series Photometry of the Open Cluster NGC 6791
• Assessment of Systematic Chromatic Errors that Impact Sub-1% Photometric Precision in Large-Area Sky Surveys
• The Overlooked Potential of Generalized Linear Models in Astronomy-III: Bayesian Negative Binomial Regression and Globular Cluster Populations
• Robust PCA and MIC statistics of baryons in early mini-haloes
• The Overlooked Potential of Generalized Linear Models in Astronomy-II: Gamma regression and photometric redshifts
• Reconstructing the galaxy density field with photometric redshifts: I. Methodology and validation on stellar mass functions
• Searching High Redshift Large-Scale Structures: Photometry of Four Fields Around Quasar Pairs at z~1
• The first analytical expression to estimate photometric redshifts suggested by a machine
• Is the cluster environment quenching the Seyfert activity in elliptical and spiral galaxies?
• Sacrificing information for the greater good: how to select photometric bands for optimal accuracy
• Photometric Redshift Biases from Galaxy Evolution
• Photo-z-SQL: integrated, flexible photometric redshift computation in a database
• Using the 2-MASS Photometric Redshift Survey to optimize LIGO Follow-Up Observations

Cited by in corpus (47)

• Weak lensing for precision cosmology
• The Southern Photometric Local Universe Survey (S-PLUS): improved SEDs, morphologies and redshifts with 12 optical filters
• Photometric redshifts from SDSS images using a Convolutional Neural Network
• The MUSE Hubble Ultra Deep Field Survey: II. Spectroscopic redshifts and comparisons to color selections of high-redshift galaxies
• Surveying the reach and maturity of machine learning and artificial intelligence in astronomy
• Photometric redshifts for the next generation of deep radio continuum surveys - I: Template fitting
• Horizon-AGN virtual observatory - 1. SED-fitting performance and forecasts for future imaging surveys
• PS1-STRM: Neural network source classification and photometric redshift catalogue for PS1 $3\pi$ DR1
• Photometry of high-redshift blended galaxies using deep learning
• Gaia GraL: Gaia DR2 Gravitational Lens Systems. III. A systematic blind search for new lensed systems
• Conditional Density Estimation Tools in Python and R with Applications to Photometric Redshifts and Likelihood-Free Cosmological Inference
• Anomaly Detection in the Open Supernova Catalog
• XMMPZCAT: A catalogue of photometric redshifts for X-ray sources
• A probabilistic approach to emission-line galaxy classification
• Return of the features. Efficient feature selection and interpretation for photometric redshifts
• The MUSE Hubble Ultra Deep Field Survey: III. Testing photometric redshifts to 30th magnitude
• A Galaxy Redshift Survey near HST/COS AGN Sight Lines
• Youth analysis of near infrared spectra of young low-mass stars and brown dwarfs
• 21st Century Statistical and Computational Challenges in Astrophysics
• Comparison of Observed Galaxy Properties with Semianalytic Model Predictions using Machine Learning
• Augmenting machine learning photometric redshifts with Gaussian mixture models
• Bayesian photometric redshifts of blended sources
• Photometric redshift estimation of galaxies in the DESI Legacy Imaging Surveys
• Estimating Photometric Redshifts for X-ray sources in the X-ATLAS field, using machine-learning techniques
• A case study of hurdle and generalized additive models in astronomy: the escape of ionizing radiation
• PhotoWeb redshift: boosting photometric redshift accuracy with large spectroscopic surveys
• CD-split and HPD-split: efficient conformal regions in high dimensions
• Estimation of Photometric Redshifts. I. Machine Learning Inference for Pan-STARRS1 Galaxies Using Neural Networks
• Optimising a magnitude-limited spectroscopic training sample for photometric classification of supernovae
• Improving Photometric Redshift Estimates with Training Sample Augmentation
• Photo-z-SQL: integrated, flexible photometric redshift computation in a database
• Improved Tomographic Binning of 3x2pt Lens Samples: Neural Network Classifiers and Optimal Bin Assignments
• Degradation analysis in the estimation of photometric redshifts from non-representative training sets
• The many flavours of photometric redshifts
• The effect of emission lines on the performance of photometric redshift estimation algorithms
• The sensitivity of GPz estimates of photo-z posterior PDFs to realistically complex training set imperfections
• Hawaii Two-0: High-redshift galaxy clustering and bias
• Dark Energy Survey Deep Field photometric redshift performance and training incompleteness assessment
• Probing the origins. I. Generalised Additive Model inference of birth radii for Milky Way stars in the solar vicinity
• Photometric Selection of type 1 Quasars in the XMM-LSS Field with Machine Learning and the Disk-Corona Connection
• Euclid: Photometric redshift calibration with self-organising maps
• A lognormal luminosity function for SWIRE in flat cosmology
• The S-PLUS Fornax Project (S+FP): An extragalactic catalog covering $\sim$ 5 virial radii around NGC 1399 with galaxy properties
• Improving Photometric Redshifts by Merging Probability Density Functions from Template-Based and Machine Learning Algorithms
• Re-calibrating Photometric Redshift Probability Distributions Using Feature-space Regression
• TOPz: Photometric redshifts using template fitting applied to the GAMA survey
• UV bright red-sequence galaxies: a comparative study between UV upturn and UV weak systems