Generalized massive optimal data compression

Abstract

Data compression has become one of the cornerstones of modern astronomical data analysis, with the vast majority of analyses compressing large raw datasets down to a manageable number of informative summaries. In this paper we provide a general procedure for optimally compressing $N$ data down to $n$ summary statistics, where $n$ is equal to the number of parameters of interest. We show that compression to the score function -- the gradient of the log-likelihood with respect to the parameters -- yields $n$ compressed statistics that are optimal in the sense that they preserve the Fisher information content of the data. Our method generalizes earlier work on linear Karhunen-Loéve compression for Gaussian data whilst recovering both lossless linear compression and quadratic estimation as special cases when they are optimal. We give a unified treatment that also includes the general non-Gaussian case as long as mild regularity conditions are satisfied, producing optimal non-linear summary statistics when appropriate. As a worked example, we derive explicitly the $n$ optimal compressed statistics for Gaussian data in the general case where both the mean and covariance depend on the parameters.

5 pages; updated to MNRAS Letters accepted version (3 Apr 2018)

References in corpus (6)

• The star formation histories of galaxies in the Sloan Digital Sky Survey
• Constraining the thick disc formation scenario of the Milky Way
• Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
• Massive data compression for parameter-dependent covariance matrices
• Maximal compression of the redshift space galaxy power spectrum and bispectrum
• An investigation into the Multiple Optimised Parameter Estimation and Data compression algorithm

Cited by in corpus (62)

• The frontier of simulation-based inference
• The Quijote simulations
• Mining gold from implicit models to improve likelihood-free inference
• Massive optimal data compression and density estimation for scalable, likelihood-free inference in cosmology
• Parameter constraints from cross-correlation of CMB lensing with galaxy clustering
• Likelihood-free inference with neural compression of DES SV weak lensing map statistics
• Probing Parity-Violation with the Four-Point Correlation Function of BOSS Galaxies
• Nuisance hardened data compression for fast likelihood-free inference
• Automatic physical inference with information maximising neural networks
• Mining for Dark Matter Substructure: Inferring subhalo population properties from strong lenses with machine learning
• Bayesian optimisation for likelihood-free cosmological inference
• Fewer Mocks and Less Noise: Reducing the Dimensionality of Cosmological Observables with Subspace Projections
• Cosmology Without Windows: Quadratic Estimators for the Galaxy Power Spectrum
• Going Beyond the Galaxy Power Spectrum: an Analysis of BOSS Data with Wavelet Scattering Transforms
• Cosmology Without Window Functions: Cubic Estimators for the Galaxy Bispectrum
• Fast and Credible Likelihood-Free Cosmology with Truncated Marginal Neural Ratio Estimation
• Enhancing BOSS bispectrum cosmological constraints with maximal compression
• ENCORE: An $\mathcal{O}(N_{\mathrm{g}}^2)$ Estimator for Galaxy $N$-Point Correlation Functions
• Lossless, Scalable Implicit Likelihood Inference for Cosmological Fields
• Data compression in cosmology: A compressed likelihood for Planck data
• The Cosmic Graph: Optimal Information Extraction from Large-Scale Structure using Catalogues
• Cosmic Shear: Inference from Forward Models
• MUSE: Marginal Unbiased Score Expansion and Application to CMB Lensing
• The sum of the masses of the Milky Way and M31: a likelihood-free inference approach
• Likelihood Non-Gaussianity in Large-Scale Structure Analyses
• Euclid: The reduced shear approximation and magnification bias for Stage IV cosmic shear experiments
• On the accuracy and precision of correlation functions and field-level inference in cosmology
• Measurement of the B-band Galaxy Luminosity Function with Approximate Bayesian Computation
• KiDS-SBI: Simulation-based inference analysis of KiDS-1000 cosmic shear
• Wavelet Moments for Cosmological Parameter Estimation
• Quijote-PNG: Quasi-maximum likelihood estimation of Primordial Non-Gaussianity in the non-linear dark matter density field
• A First Detection of the Connected 4-Point Correlation Function of Galaxies Using the BOSS CMASS Sample
• The PAU Survey: Measurement of Narrow-band galaxy properties with Approximate Bayesian Computation
• 21st Century Statistical and Computational Challenges in Astrophysics
• Effective LHC measurements with matrix elements and machine learning
• On cosmological bias due to the magnification of shear and position samples in modern weak lensing analyses
• Probing massive neutrinos with the Minkowski functionals of the galaxy distribution
• Unbiased likelihood-free inference of the Hubble constant from light standard sirens
• Modal compression of the redshift-space galaxy bispectrum
• Data Compression and Inference in Cosmology with Self-Supervised Machine Learning
• A simulation-based inference pipeline for cosmic shear with the Kilo-Degree Survey
• Extreme data compression while searching for new physics
• Cosmological Parameter Estimation and Inference using Deep Summaries
• Quijote-PNG: The Information Content of the Halo Mass Function
• GEOMAX: beyond linear compression for 3pt galaxy clustering statistics
• Bayesian Control Variates for optimal covariance estimation with pairs of simulations and surrogates
• Quijote-PNG: Quasi-maximum likelihood estimation of Primordial Non-Gaussianity in the non-linear halo density field
• Catalog-free modeling of galaxy types in deep images: Massive dimensional reduction with neural networks
• Measuring the thermal and ionization state of the low-$z$ IGM using likelihood free inference
• The impact of signal-to-noise, redshift, and angular range on the bias of weak lensing 2-point functions
• Full-waveform earthquake source inversion using simulation-based inference
• Comparison of Bayesian inference methods using the Loreli II database of hydro-radiative simulations of the 21-cm signal
• Simulation-based inference has its own Dodelson-Schneider effect (but it knows that it does)
• Hybrid summary statistics: neural weak lensing inference beyond the power spectrum
• Simulation-based inference of Bayesian hierarchical models while checking for model misspecification
• A data compression and optimal galaxy weights scheme for Dark Energy Spectroscopic Instrument and weak lensing datasets
• Effective experience rating for large insurance portfolios via surrogate modeling
• A Python compressed low-$\ell$ Planck likelihood for temperature and polarization
• Fisher information flow in artificial neural networks
• Quijote PNG: The information content of the halo power spectrum and bispectrum
• Diagnosing Systematic Effects Using the Inferred Initial Power Spectrum
• Optimal data compression for Lyman-$\alpha$ forest cosmology