Data-Space Validation of High-Dimensional Models by Comparing Sample Quantiles
arXiv:2402.00930 · doi:10.3847/1538-4365/ad8ebd
Abstract
We present a simple method for assessing the predictive performance of high-dimensional models directly in data space when only samples are available. Our approach is to compare the quantiles of observables predicted by a model to those of the observables themselves. In cases where the dimensionality of the observables is large (e.g. multiband galaxy photometry), we advocate that the comparison is made after projection onto a set of principal axes to reduce the dimensionality. We demonstrate our method on a series of two-dimensional examples. We then apply it to results from a state-of-the-art generative model for galaxy photometry (pop-cosmos; arXiv:2402.00935) that generates predictions of colors and magnitudes by forward simulating from a 16-dimensional distribution of physical parameters represented by a score-based diffusion model. We validate the predictive performance of this model directly in a space of nine broadband colors. Although motivated by this specific example, we expect that the techniques we present will be broadly useful for evaluating the performance of flexible, non-parametric population models of this kind, and other settings where two sets of samples are to be compared.
21 pages, 11 figures. Accepted for publication in ApJS. Companion paper to arXiv:2402.00935
References in corpus (76)
- SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
- Array Programming with NumPy
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- LSST: from Science Drivers to Reference Design and Anticipated Data Products
- The Cosmic Evolution Survey (COSMOS) -- Overview
- Statistical applications of the multivariate skew-normal distribution
- The COSMOS2015 Catalog: Exploring the 1<z<6 Universe with half a million galaxies
- Statistics: Handle with Care, Detecting Multiple Model Components with the Likelihood Ratio Test
- Philosophy and the practice of Bayesian statistics
- Equivalence of distance-based and RKHS-based statistics in hypothesis testing
- How to Measure Galaxy Star Formation Histories II: Nonparametric Models
- Deriving Physical Properties from Broadband Photometry with Prospector: Description of the Model and a Demonstration of its Accuracy Using 129 Galaxies in the Local Universe
- The Dark Energy Survey Data Release 1
- COSMOS2020: A panchromatic view of the Universe to from two complementary catalogs
- Discovering physical concepts with neural networks
- Modelling and interpreting spectral energy distributions of galaxies with BEAGLE
- The fourth data release of the Kilo-Degree Survey: ugri imaging and nine-band optical-IR photometry over 1000 square degrees
- PCA consistency in high dimension, low sample size context
- An Older, More Quiescent Universe from Panchromatic SED Fitting of the 3D-HST Survey
- Prospects for resolving the Hubble constant tension with standard sirens
- Dark Energy Survey Year 3 Results: Photometric Data Set for Cosmology
- Distributions of Galaxy Spectral Types in the Sloan Digital Sky Survey
- Constructing A Flexible Likelihood Function For Spectroscopic Inference
- Approximate Bayesian computation with the Wasserstein distance
- Principal Component Analysis with Noisy and/or Missing Data
- Likelihood-free inference with neural compression of DES SV weak lensing map statistics
- Automated Classification of Stellar Spectra. II: Two-Dimensional Classification with Neural Networks and Principal Components Analysis
- Hot dust in Panchromatic SED Fitting: Identification of AGN and improved galaxy properties
- Dimensionality Reduction of SDSS Spectra with Variational Autoencoders
- SPECULATOR: Emulating stellar population synthesis for fast and accurate galaxy spectra and photometry
- BayeSED: A General Approach to Fitting the Spectral Energy Distribution of Galaxies
- Dark Energy Survey Year 1 Results: Joint Analysis of Galaxy Clustering, Galaxy Lensing, and CMB Lensing Two-point Functions
- Quasi-Periodic Oscillations and broadband variability in short magnetar bursts
- Reducing the Dimensionality of Data: Locally Linear Embedding of Sloan Galaxy Spectra
- Information Gains from Cosmic Microwave Background Experiments
- Planck confronts large scale structure: methods to quantify discordance
- Quantifying Tensions between CMB and Distance Datasets in Models with Free Curvature or Lensing Amplitude
- On the Application of Bayesian Leave-One-Out Cross-Validation to Exoplanet Atmospheric Analysis
- Consistency tests in cosmology using relative entropy
- Constraining the dark energy equation of state using Bayes theorem and the Kullback-Leibler divergence
- General framework for cosmological dark matter bounds using -body simulations
- How much a galaxy knows about its large-scale environment?: An information theoretic perspective
- Two-Component Structure of the Hbeta Broad-Line Region in Quasars. I. Evidence from Spectral Principal Component Analysis
- Global regularities in integrated galaxy spectra
- Searching for Narrow Emission Lines in X-ray Spectra: Computation and Methods
- First semi-empirical test of the white dwarf mass-radius relationship using a single white dwarf via astrometric microlensing
- Information Gains from Cosmological Probes
- Forward modeling of galaxy populations for cosmological redshift distribution inference
- Filling the gaps: Gaussian mixture models from noisy, truncated or incomplete samples
- Hierarchical Bayesian inference of photometric redshifts with stellar population synthesis models
- A robust estimator of mutual information for deep learning interpretability
- Gaussianisation for fast and accurate inference from cosmological data
- pop-cosmos: A comprehensive picture of the galaxy population from COSMOS data
- Similarity of samples and trimming
- Comparing cosmic web classifiers using information theory
- Model Assessment Tools for a Model False World
- Cosmic web-type classification using decision theory
- Locally linear embedding: dimension reduction of massive protostellar spectra
- Sufficiency of a Gaussian power spectrum likelihood for accurate cosmology from upcoming weak lensing surveys
- Principal component analysis of the Spitzer IRS spectra of ultraluminous infrared galaxies
- Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
- The PAU Survey: A Forward Modeling Approach for Narrow-band Imaging
- Explaining dark matter halo density profiles with neural networks
- PopSED: Population-Level Inference for Galaxy Properties from Broadband Photometry with Neural Density Estimation
- A Machine Learning Approach to Predict Missing Flux Densities in Multi-band Galaxy Surveys
- Detecting Unspecified Structure in Low-Count Images
- Simulation-based inference of deep fields: galaxy population model and redshift distributions
- Integrated cosmological probes: Concordance quantified
- Model Breaking Measure for Cosmological Surveys
- Single parameter galaxy classification: The Principal Curve through the multi-dimensional space of galaxy properties
- A representation learning approach to probe for dynamical dark energy in matter power spectra
- Stress testing the dark energy equation of state imprint on supernova data
- qrpca: A Package for Fast Principal Component Analysis with GPU Acceleration
- Deep learning insights into non-universality in the halo mass function
- Principal component analysis of IUE galaxy spectra
- yonder: A python package for data denoising and reconstruction
Cited by in corpus (4)
- pop-cosmos: Insights from generative modeling of a deep, infrared-selected galaxy population
- pop-cosmos: Star formation over 12 Gyr from generative modelling of a deep infrared-selected galaxy catalogue
- GalSBI-SPS: a stellar population synthesis-based galaxy population model for cosmology and galaxy evolution applications
- GalSBI: Phenomenological galaxy population model for cosmology using simulation-based inference