Approximating photo-$z$ PDFs for large surveys

Abstract

Modern galaxy surveys produce redshift probability density functions (PDFs) in addition to traditional photometric redshift (photo-$z$) point estimates. However, the storage of photo-$z$ PDFs may present a challenge with increasingly large catalogs, as we face a trade-off between the accuracy of subsequent science measurements and the limitation of finite storage resources. This paper presents $\text{qp}$, a Python package for manipulating parametrizations of 1-dimensional PDFs, as suitable for photo-$z$ PDF compression. We use $\text{qp}$ to investigate the performance of three simple PDF storage formats (quantiles, samples, and step functions) as a function of the number of stored parameters on two realistic mock datasets, representative of upcoming surveys with different data qualities. We propose some best practices for choosing a photo-$z$ PDF approximation scheme and demonstrate the approach on a science case using performance metrics on both ensembles of individual photo-$z$ PDFs and an estimator of the overall redshift distribution function. We show that both the properties of the set of PDFs we wish to approximate and the chosen fidelity metric(s) affect the optimal parametrization. Additionally, we find that quantiles and samples outperform step functions, and we encourage further consideration of these formats for PDF approximation.

References in corpus (7)

• The NumPy array: a structure for efficient numerical computation
• METAPHOR: A machine learning based method for the probability density estimation of photometric redshifts
• Photometric Redshifts with the LSST: Evaluating Survey Observing Strategies
• Searching for galaxy clusters in the Kilo-Degree Survey
• Cross-correlation of weak lensing and gamma rays: implications for the nature of dark matter
• Sparse Representation of Photometric Redshift PDFs: Preparing for Petascale Astronomy
• The X-ray Binary Population of the Nearby Dwarf Starburst Galaxy IC 10: Variable and Transient X-ray Sources

Cited by in corpus (18)

• Non-parametric Star Formation History Reconstruction with Gaussian Processes I: Counting Major Episodes of Star Formation
• Photometric Redshifts for Next-Generation Surveys
• Phenotypic redshifts with self-organizing maps: A novel method to characterize redshift distributions of source galaxies for weak lensing
• Conditional Density Estimation Tools in Python and R with Applications to Photometric Redshifts and Likelihood-Free Cosmological Inference
• ADDGALS: Simulated Sky Catalogs for Wide Field Galaxy Surveys
• Photo-z outlier self-calibration in weak lensing surveys
• Photometric redshifts with machine learning, lights and shadows on a complex data science use case
• The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC): Selection of a performance metric for classification probabilities balancing diverse science goals
• A Unified Catalog-level Reanalysis of Stage-III Cosmic Shear Surveys
• Photometric Redshift Uncertainties in Weak Gravitational Lensing Shear Analysis: Models and Marginalization
• PICZL: Image-based Photometric Redshifts for AGN
• Stress testing the dark energy equation of state imprint on supernova data
• Gaussian Mixture Models for Blended Photometric Redshifts
• The sensitivity of GPz estimates of photo-z posterior PDFs to realistically complex training set imperfections
• A novel Deep Learning approach for one-step Conformal Prediction approximation
• Mapping Variations of Redshift Distributions with Probability Integral Transforms
• Simultaneous Estimation of Large-Scale Structure and Milky Way Dust Extinction from Galaxy Surveys
• ColdPress: Efficient Quantile-Based Compression of Photometric Redshift PDFs