TPZ : Photometric redshift PDFs and ancillary information by using prediction trees and random forests

Abstract

With the growth of large photometric surveys, accurately estimating photometric redshifts, preferably as a probability density function (PDF), and fully understanding the implicit systematic uncertainties in this process has become increasingly important. In this paper, we present a new, publicly available, parallel, machine learning algorithm that generates photometric redshift PDFs by using prediction trees and random forest techniques, which we have named TPZ. This new algorithm incorporates measurement errors into the calculation while also dealing efficiently with missing values in the data. In addition, our implementation of this algorithm provides supplementary information regarding the data being analyzed, including unbiased estimates of the accuracy of the technique without resorting to a validation data set, identification of poor photometric redshift areas within the parameter space occupied by the spectroscopic training data, a quantification of the relative importance of the variables used to construct the PDF, and a robust identification of outliers. This extra information can be used to optimally target new spectroscopic observations and to improve the overall efficacy of the redshift estimation. We have tested TPZ on galaxy samples drawn from the SDSS main galaxy sample and from the DEEP2 survey, obtaining excellent results in each case. We also have tested our implementation by participating in the PHAT1 project, which is a blind photometric redshift contest, finding that TPZ performs comparable to if not better than other empirical photometric redshift algorithms. Finally, we discuss the various parameters that control the operation of TPZ, the specific limitations of this approach and an application of photometric redshift PDFs.

21 pages, 15 figures, Accepted for publication in MNRAS. TPZ code at http://lcdm.astro.illinois.edu/research/TPZ.html

References in corpus (11)

• The Zurich Extragalactic Bayesian Redshift Analyzer (ZEBRA) and its first application: COSMOS
• A Spectroscopic Survey of Redshift 1.4<z<3.0 Galaxies in the Goods-North Field: Survey Description, Catalogs, and Properties
• Estimating the Redshift Distribution of Faint Galaxy Samples
• A Galaxy Photometric Redshift Catalog for the Sloan Digital Sky Survey Data Release 6
• Precision photometric redshift calibration for galaxy-galaxy weak lensing
• Robust Machine Learning Applied to Astronomical Datasets III: Probabilistic Photometric Redshifts for Galaxies and Quasars in the SDSS and GALEX
• New Approaches To Photometric Redshift Prediction Via Gaussian Process Regression In The Sloan Digital Sky Survey
• Photometric redshifts with Quasi Newton Algorithm (MLPQNA). Results in the PHAT1 contest
• Extended Photometry for the DEEP2 Galaxy Redshift Survey: A Testbed for Photometric Redshift Experiments
• On estimating redshift and luminosity distributions in photometric redshift surveys
• Two novel approaches for photometric redshift estimation based on SDSS and 2MASS databases

Cited by in corpus (172)

• Machine learning and the physical sciences
• The Dark Energy Survey: more than dark energy - an overview
• The Sloan Digital Sky Survey Quasar Catalog: Sixteenth Data Release
• Photometric Redshifts for Hyper Suprime-Cam Subaru Strategic Program Data Release 1
• The CosmoVerse White Paper: Addressing observational tensions in cosmology with systematics and fundamental physics
• Star-galaxy Classification Using Deep Convolutional Neural Networks
• Dark Energy Survey Year 1 Results: Redshift distributions of the weak lensing source galaxies
• ANNz2 - photometric redshift and probability distribution function estimation using machine learning
• The Clustering of DESI-like Luminous Red Galaxies Using Photometric Redshifts
• Photometric redshifts from SDSS images using a Convolutional Neural Network
• Photometric redshift analysis in the Dark Energy Survey Science Verification data
• The Dust-to-Gas and Dust-to-Metals Ratio in Galaxies from z=0-6
• Cosmology from Cosmic Shear with DES Science Verification Data
• Photometric redshift estimation via deep learning
• Photometric Redshift with Bayesian Priors on Physical Properties of Galaxies
• Redshift distributions of galaxies in the DES Science Verification shear catalogue and implications for weak lensing
• Surveying the reach and maturity of machine learning and artificial intelligence in astronomy
• The Complete Calibration of the Color-Redshift Relation (C3R2) Survey: Survey Overview and Data Release 1
• Weak-lensing mass calibration of redMaPPer galaxy clusters in Dark Energy Survey Science Verification data
• The Complete Calibration of the Color-Redshift Relation (C3R2) Survey: Analysis and Data Release 2
• DNF - Galaxy photometric redshift by Directional Neighbourhood Fitting
• SOMz: photometric redshift PDFs with self organizing maps and random atlas
• CMB lensing tomography with the DES Science Verification galaxies
• GPz: Non-stationary sparse Gaussian processes for heteroscedastic uncertainty estimation in photometric redshifts
• Photometric Redshifts for Next-Generation Surveys
• Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing
• Spectroscopic Needs for Imaging Dark Energy Experiments: Photometric Redshift Training and Calibration
• All-purpose, all-sky photometric redshifts for the Legacy Imaging Surveys Data Release 8
• Galaxy clustering, photometric redshifts and diagnosis of systematics in the DES Science Verification data
• Galaxy clustering in the DESI Legacy Survey and its imprint on the CMB
• Photometric redshifts for the next generation of deep radio continuum surveys - I: Template fitting
• Dark Energy Survey Year 1 Results: Cross-Correlation Redshifts - Methods and Systematics Characterization
• Evaluation of probabilistic photometric redshift estimation approaches for The Rubin Observatory Legacy Survey of Space and Time (LSST)
• Using neural networks to estimate redshift distributions. An application to CFHTLenS
• A Sparse Gaussian Process Framework for Photometric Redshift Estimation
• Catalogues of Active Galactic Nuclei From Gaia and unWISE Data
• Cosmology from large scale galaxy clustering and galaxy-galaxy lensing with Dark Energy Survey Science Verification data
• Precise photometric redshifts with a narrow-band filter set: The PAU Survey at the William Herschel Telescope
• Photometric Redshift Estimation with a Convolutional Neural Network: NetZ
• Anomaly detection for machine learning redshifts applied to SDSS galaxies
• Mapping and simulating systematics due to spatially-varying observing conditions in DES Science Verification data
• Redshift inference from the combination of galaxy colors and clustering in a hierarchical Bayesian model
• Feature importance for machine learning redshifts applied to SDSS galaxies
• Exhausting the Information: Novel Bayesian Combination of Photometric Redshift PDFs
• Machine Learning and Cosmological Simulations II: Hydrodynamical Simulations
• Joint Measurement of Lensing-Galaxy Correlations Using SPT and DES SV Data
• Cross-correlation of gravitational lensing from DES Science Verification data with SPT and Planck lensing
• Photometric redshifts for the next generation of deep radio continuum surveys - II. Gaussian processes and hybrid estimates
• Deep Transfer Learning for Star Cluster Classification: I. Application to the PHANGS-HST Survey
• Hierarchical Bayesian inference of galaxy redshift distributions from photometric surveys
• Morpho-z: improving photometric redshifts with galaxy morphology
• Disentangling the AGN and Star-Formation connection using XMM-Newton
• Machine Learning Classification of SDSS Transient Survey Images
• Selection biases in empirical p(z) methods for weak lensing
• A Hybrid Ensemble Learning Approach to Star-Galaxy Classification
• Improving Photometric Redshift Estimation using GPz: size information, post processing and improved photometry
• Machine Learning and Cosmological Simulations I: Semi-Analytical Models
• 2dFLenS and KiDS: Determining source redshift distributions with cross-correlations
• Intrinsic alignments of group and cluster galaxies in photometric surveys
• A machine learning approach to galaxy properties: joint redshift-stellar mass probability distributions with Random Forest
• Conditional Density Estimation Tools in Python and R with Applications to Photometric Redshifts and Likelihood-Free Cosmological Inference
• Probing the Nature of High Redshift Weak Emission Line Quasars: A Young Quasar with a Starburst Host Galaxy
• Redshift inference from the combination of galaxy colors and clustering in a hierarchical Bayesian model $-$ Application to realistic $N$-body simulations
• Young Galaxy Candidates in the Hubble Frontier Fields. II. MACS,J0416-2403
• The Extremely Luminous Quasar Survey (ELQS) in the SDSS footprint I.: Infrared Based Candidate Selection
• Data augmentation for machine learning redshifts applied to SDSS galaxies
• Photometric Redshift Calibration Requirements for WFIRST Weak Lensing Cosmology: Predictions from CANDELS
• The relation between AGN type and host galaxy properties
• Weak Lensing Tomographic Redshift Distribution Inference for the Hyper Suprime-Cam Subaru Strategic Program three-year shape catalogue
• XMMPZCAT: A catalogue of photometric redshifts for X-ray sources
• Analysis of a Custom Support Vector Machine for Photometric Redshift Estimation and the Inclusion of Galaxy Shape Information
• J-PLUS: On the identification of new cluster members in the double galaxy cluster A2589 & A2593 using PDFs
• Weak lensing magnification in the Dark Energy Survey Science Verification Data
• Data-driven, interpretable photometric redshifts trained on heterogeneous and unrepresentative data
• The redshift distribution of cosmological samples: a forward modeling approach
• Improving the reliability of photometric redshift with machine learning
• The Extremely Luminous Quasar Survey in the Pan-STARRS 1 Footprint (PS-ELQS)
• J-PLUS: measuring $\mathrm{H}\alpha$ emission line fluxes in the nearby universe
• The Fundamental Relation between Halo Mass and Galaxy Group Properties
• Photometric redshifts with machine learning, lights and shadows on a complex data science use case
• Stacking for machine learning redshifts applied to SDSS galaxies
• Hierarchical modeling and statistical calibration for photometric redshifts
• Galaxy bias from the Dark Energy Survey Science Verification data: combining galaxy density maps and weak lensing maps
• Environmental dependence of the galaxy stellar mass function in the Dark Energy Survey Science Verification Data
• The first analytical expression to estimate photometric redshifts suggested by a machine
• Assessing the photometric redshift precision of the S-PLUS survey: the Stripe-82 as a test-case
• Statistical analysis of probability density functions for photometric redshifts through the KiDS-ESO-DR3 galaxies
• Simultaneous Estimation of Photometric Redshifts and SED Parameters: Improved Techniques and a Realistic Error Budget
• DES Science Portal: Computing Photometric Redshifts
• GAlaxy Light profile convolutional neural NETworks (GaLNets). I. fast and accurate structural parameters for billion galaxy samples
• A Unified Framework for Constructing, Tuning and Assessing Photometric Redshift Density Estimates in a Selection Bias Setting
• Effect of training characteristics on object classification: an application using Boosted Decision Trees
• Hierarchical Bayesian inference of photometric redshifts with stellar population synthesis models
• What to expect from dynamical modelling of cluster haloes II. Investigating dynamical state indicators with Random Forest
• Photometric Classification of quasars from RCS-2 using Random Forest
• Exploring Photometric Redshifts as an Optimization Problem: An Ensemble MCMC and Simulated Annealing-Driven Template-Fitting Approach
• Inferring physical properties of galaxies from their emission line spectra
• The ALHAMBRA survey: an empirical estimation of the cosmic variance for merger fraction studies based on close pairs
• Comparison of Observed Galaxy Properties with Semianalytic Model Predictions using Machine Learning
• The ALHAMBRA survey : $B-$band luminosity function of quiescent and star-forming galaxies at $0.2\le z<1$ by PDF analysis
• Photometric Redshift Uncertainties in Weak Gravitational Lensing Shear Analysis: Models and Marginalization
• Augmenting machine learning photometric redshifts with Gaussian mixture models
• Bayesian photometric redshifts of blended sources
• Hybrid photometric redshifts for sources in the COSMOS and XMM-LSS fields
• Sparse Representation of Photometric Redshift PDFs: Preparing for Petascale Astronomy
• Estimating Photometric Redshifts for X-ray sources in the X-ATLAS field, using machine-learning techniques
• Testing the evolutionary pathways of galaxies and their supermassive black holes and the impact of feedback from Active Galactic Nuclei via large multiwavelength datasets
• Estimating redshift distributions using Hierarchical Logistic Gaussian processes
• Photometric Redshift Estimation for Quasars by Integration of KNN and SVM
• A Composite Likelihood Approach for Inference under Photometric Redshift Uncertainty
• Photometric Redshift Estimation with Galaxy Morphology using Self-Organizing Maps
• Galaxy clustering with photometric surveys using PDF redshift information
• Photometric redshifts for the Pan-STARRS1 survey
• Measuring Linear and Non-linear Galaxy Bias Using Counts-in-Cells in the Dark Energy Survey Science Verification Data
• Sacrificing information for the greater good: how to select photometric bands for optimal accuracy
• TOPz: Photometric redshifts for J-PAS
• Galaxy bias from galaxy-galaxy lensing in the DES Science Verification Data
• Mixture Models for Photometric Redshifts
• GeneticKNN: A Weighted KNN Approach Supported by Genetic Algorithm for Photometric Redshift Estimation of Quasars
• CD-split and HPD-split: efficient conformal regions in high dimensions
• The 2-degree Field Lensing Survey: photometric redshifts from a large new training sample to r<19.5
• Improving Photometric Redshift Estimates with Training Sample Augmentation
• Machine-learning computation of distance modulus for local galaxies
• Reliable Photometric Membership (RPM) of Galaxies in Clusters. I. A Machine Learning Method and its Performance in the Local Universe
• Machine learning synthetic spectra for probabilistic redshift estimation: SYTH-Z
• Galaxy blending effects in deep imaging cosmic shear probes of cosmology
• A machine learning approach to infer the accreted stellar mass fractions of central galaxies in the TNG100 simulation
• Self-consistent redshift estimation using correlation functions without a spectroscopic reference sample
• PICZL: Image-based Photometric Redshifts for AGN
• Correcting cosmological parameter biases for all redshift surveys induced by estimating and reweighting redshift distributions
• Photo-z Quality Cuts and their Effect on the Measured Galaxy Clustering
• A search for X-ray absorbed sources in the 3XMM catalogue, using photometric redshifts and Bayesian spectral fits
• The PAU Survey: Photometric redshift estimation in deep wide fields
• Merged or monolithic? Using machine-learning to reconstruct the dynamical history of simulated star clusters
• Euclid preparation. XXXI. The effect of the variations in photometric passbands on photometric-redshift accuracy
• Improved Tomographic Binning of 3x2pt Lens Samples: Neural Network Classifiers and Optimal Bin Assignments
• Inferring redshift and galaxy properties via a multi-task neural net with probabilistic outputs: An application to simulated MOONS spectra
• Tuning target selection algorithms to improve galaxy redshift estimates
• Gaussian Mixture Models for Blended Photometric Redshifts
• Learning Spectral Templates for Photometric Redshift Estimation from Broadband Photometry
• The many flavours of photometric redshifts
• Degradation analysis in the estimation of photometric redshifts from non-representative training sets
• A Machine Learning Approach to Correct for Mass Resolution Effects in Simulated Halo Clustering Statistics
• Improved photometric redshifts with colour-constrained galaxy templates for future wide-area surveys
• Dark Energy Survey Deep Field photometric redshift performance and training incompleteness assessment
• DES Science Portal: Creating Science-Ready Catalogs
• Rejection criteria based on outliers in the KiDS photometric redshifts and PDF distributions derived by machine learning
• The Sensitivity of the Redshift Distribution to Galaxy Demographics
• Euclid: Photometric redshift calibration with self-organising maps
• CLAP. I. Resolving miscalibration for deep learning-based galaxy photometric redshift estimation
• SpecPT (Spectroscopy Pre-trained Transformer) Model for Extragalactic Spectroscopy: I. Architecture and Automated Redshift Measurement
• On the Clustering of Compact Galaxy Pairs in Dark Matter Haloes
• Three-Dimensional Reconstruction of Weak Lensing Mass Maps with a Sparsity Prior. I. Cluster Detection
• Euclid Preparation: XIV. The Complete Calibration of the Color-Redshift Relation (C3R2) Survey: Data Release 3
• Harnessing the XMM-Newton data: X-ray spectral modelling of 4XMM-DR11 detections and 4XMM-DR11s sources
• Reconstructing AGN X-ray spectral parameter distributions with Bayesian methods I: Spectral analysis
• Z-Sequence: Photometric redshift predictions for galaxy clusters with sequential random k-nearest neighbours
• Pz Cats: Photometric redshift catalogs based on DES Y3 BAO sample
• Contaminating Electromagnetic Transients in LISA Gravitational Wave Localization Volumes. I: The Intrinsic Rates
• CSST Slitless Spectra: Target Detection and Classification with YOLO
• AGN obscuration in optical and X-rays: Host properties and the interplay of nuclear and galactic gas and dust in a combined SDSS-XMM sample
• Non-Sequential Neural Network for Simultaneous, Consistent Classification and Photometric Redshifts of OTELO Galaxies
• Photometric Redshift Estimation with Convolutional Neural Networks and Galaxy Images: A Case Study of Resolving Biases in Data-Driven Methods
• Photometric redshifts for the S-PLUS Survey: is machine learning up to the task?
• Using a Neural Network Classifier to Select Galaxies with the Most Accurate Photometric Redshifts
• TOPz: Photometric redshifts using template fitting applied to the GAMA survey
• Photometric Redshift Estimation Using Scaled Ensemble Learning
• The Redshifts from 122 Bands: Comparative Redshift Forecast for Low-Resolution Spectra from SPHEREx and 7-Dimensional Sky Survey (7DS)
• A catalog of photometric redshift and the distribution of broad galaxy morphologies
• DRAGNs in the Forest: Identifying Artifacts with Random Forest Models in the VLASS DRAGNs Catalog
• Identification of high-redshift X-ray active galactic nuclei in the 4XMM-DR11 serendipitous catalogue using DES data: Comparative analysis with optically-selected QSOs
• New Metric Formulas that Include Measurement Errors in Machine Learning for Natural Sciences