Generalized massive optimal data compression
arXiv:1712.00012 · doi:10.1093/mnrasl/sly029
Abstract
Data compression has become one of the cornerstones of modern astronomical data analysis, with the vast majority of analyses compressing large raw datasets down to a manageable number of informative summaries. In this paper we provide a general procedure for optimally compressing data down to summary statistics, where is equal to the number of parameters of interest. We show that compression to the score function -- the gradient of the log-likelihood with respect to the parameters -- yields compressed statistics that are optimal in the sense that they preserve the Fisher information content of the data. Our method generalizes earlier work on linear Karhunen-Loéve compression for Gaussian data whilst recovering both lossless linear compression and quadratic estimation as special cases when they are optimal. We give a unified treatment that also includes the general non-Gaussian case as long as mild regularity conditions are satisfied, producing optimal non-linear summary statistics when appropriate. As a worked example, we derive explicitly the optimal compressed statistics for Gaussian data in the general case where both the mean and covariance depend on the parameters.
arXiv comment5 pages; updated to MNRAS Letters accepted version (3 Apr 2018)
References in corpus (25)
- Karhunen-Loeve eigenvalue problems in cosmology: how should we tackle large data sets?
- The Complete Star Formation History of the Universe
- Estimating the Power Spectrum of the Cosmic Microwave Background
- How to measure CMB power spectra without losing information
- The star formation histories of galaxies in the Sloan Digital Sky Survey
- Massive Lossless Data Compression and Multiple Parameter Estimation from Galaxy Spectra
- Radical Compression of Cosmic Microwave Background Data
- Constraining the thick disc formation scenario of the Milky Way
- Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
- Approximate Bayesian Computation for Astronomical Model Analysis: A Case Study in Galaxy Demographics and Morphological Transformation at High Redshift
- Likelihood-Free Cosmological Inference with Type Ia Supernovae: Approximate Bayesian Computation for a Complete Treatment of Uncertainty
- Approximate Bayesian Computation for Forward Modeling in Cosmology
- cosmoabc: Likelihood-free inference via Population Monte Carlo Approximate Bayesian Computation
- Recovering Physical Parameters from Galaxy Spectra Using MOPED
- A new model to predict weak-lensing peak counts II. Parameter constraint strategies
- Massive data compression for parameter-dependent covariance matrices
- Fast identification of transits from light-curves
- Maximal compression of the redshift space galaxy power spectrum and bispectrum
- A New Method to Measure the Post-Reionization Ionizing Background from the Joint Distribution of Lyman- and Lyman- Forest Transmission
- Approximate Bayesian Computation in Large Scale Structure: constraining the galaxy-halo connection
- Accelerating Approximate Bayesian Computation with Quantile Regression: Application to Cosmological Redshift Distributions
- Inferring the photometric and size evolution of galaxies from image simulations
- Extreme data compression for the CMB
- Fast Parameter Estimation from the CMB Power Spectrum
- An investigation into the Multiple Optimised Parameter Estimation and Data compression algorithm
Cited by in corpus (62)
- The frontier of simulation-based inference
- The Quijote simulations
- Mining gold from implicit models to improve likelihood-free inference
- Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
- Parameter constraints from cross-correlation of CMB lensing with galaxy clustering
- Likelihood-free inference with neural compression of DES SV weak lensing map statistics
- Probing Parity-Violation with the Four-Point Correlation Function of BOSS Galaxies
- Nuisance hardened data compression for fast likelihood-free inference
- Automatic physical inference with information maximising neural networks
- Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning
- Bayesian optimisation for likelihood-free cosmological inference
- Fewer Mocks and Less Noise: Reducing the Dimensionality of Cosmological Observables with Subspace Projections
- Cosmology Without Windows: Quadratic Estimators for the Galaxy Power Spectrum
- Going Beyond the Galaxy Power Spectrum: an Analysis of BOSS Data with Wavelet Scattering Transforms
- Cosmology Without Window Functions: Cubic Estimators for the Galaxy Bispectrum
- Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimation
- Enhancing BOSS bispectrum cosmological constraints with maximal compression
- ENCORE: An Estimator for Galaxy -Point Correlation Functions
- Lossless, Scalable Implicit Likelihood Inference for Cosmological Fields
- Data compression in cosmology: A compressed likelihood for Planck data
- The Cosmic Graph: Optimal Information Extraction from Large-Scale Structure using Catalogues
- Cosmic Shear: Inference from Forward Models
- MUSE: Marginal Unbiased Score Expansion and Application to CMB Lensing
- The sum of the masses of the Milky Way and M31: a likelihood-free inference approach
- Likelihood Non-Gaussianity in Large-Scale Structure Analyses
- Euclid: The reduced shear approximation and magnification bias for Stage IV cosmic shear experiments
- On the accuracy and precision of correlation functions and field-level inference in cosmology
- Measurement of the B-band Galaxy Luminosity Function with Approximate Bayesian Computation
- KiDS-SBI: Simulation-based inference analysis of KiDS-1000 cosmic shear
- Wavelet Moments for Cosmological Parameter Estimation
- Quijote-PNG: Quasi-maximum likelihood estimation of Primordial Non-Gaussianity in the non-linear dark matter density field
- A First Detection of the Connected 4-Point Correlation Function of Galaxies Using the BOSS CMASS Sample
- The PAU Survey: Measurement of Narrow-band galaxy properties with Approximate Bayesian Computation
- 21st Century Statistical and Computational Challenges in Astrophysics
- Effective LHC measurements with matrix elements and machine learning
- On cosmological bias due to the magnification of shear and position samples in modern weak lensing analyses
- Probing massive neutrinos with the Minkowski functionals of the galaxy distribution
- Unbiased likelihood-free inference of the Hubble constant from light standard sirens
- Modal compression of the redshift-space galaxy bispectrum
- Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
- A simulation-based inference pipeline for cosmic shear with the Kilo-Degree Survey
- Extreme data compression while searching for new physics
- Cosmological Parameter Estimation and Inference using Deep Summaries
- Quijote-PNG: The Information Content of the Halo Mass Function
- GEOMAX: beyond linear compression for 3pt galaxy clustering statistics
- Bayesian Control Variates for optimal covariance estimation with pairs of simulations and surrogates
- Quijote-PNG: Quasi-maximum likelihood estimation of Primordial Non-Gaussianity in the non-linear halo density field
- Catalog-free modeling of galaxy types in deep images: Massive dimensional reduction with neural networks
- Measuring the thermal and ionization state of the low- IGM using likelihood free inference
- The impact of signal-to-noise, redshift, and angular range on the bias of weak lensing 2-point functions
- Full-waveform earthquake source inversion using simulation-based inference
- Comparison of Bayesian inference methods using the Loreli II database of hydro-radiative simulations of the 21-cm signal
- Simulation-based inference has its own Dodelson-Schneider effect (but it knows that it does)
- Hybrid summary statistics: neural weak lensing inference beyond the power spectrum
- Simulation-based inference of Bayesian hierarchical models while checking for model misspecification
- A data compression and optimal galaxy weights scheme for Dark Energy Spectroscopic Instrument and weak lensing datasets
- Effective experience rating for large insurance portfolios via surrogate modeling
- A Python compressed low- Planck likelihood for temperature and polarization
- Fisher information flow in artificial neural networks
- Quijote PNG: The information content of the halo power spectrum and bispectrum
- Diagnosing Systematic Effects Using the Inferred Initial Power Spectrum
- Optimal data compression for Lyman- forest cosmology