Extreme deconvolution: Inferring complete distribution functions from noisy, heterogeneous and incomplete observations

Abstract

We generalize the well-known mixtures of Gaussians approach to density estimation and the accompanying Expectation--Maximization technique for finding the maximum likelihood parameters of the mixture to the case where each data point carries an individual $d$-dimensional uncertainty covariance and has unique missing data properties. This algorithm reconstructs the error-deconvolved or "underlying" distribution function common to all samples, even when the individual data points are samples from different distributions, obtained by convolving the underlying distribution with the heteroskedastic uncertainty distribution of the data point and projecting out the missing data directions. We show how this basic algorithm can be extended with conjugate priors on all of the model parameters and a "split-and-merge" procedure designed to avoid local maxima of the likelihood. We demonstrate the full method by applying it to the problem of inferring the three-dimensional velocity distribution of stars near the Sun from noisy two-dimensional, transverse velocity measurements from the Hipparcos satellite.

Published in at http://dx.doi.org/10.1214/10-AOAS439 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

References in corpus (6)

• The Geneva-Copenhagen Survey of the Solar neighbourhood III. Improved distances, ages, and kinematics
• Origin and evolution of moving groups I. Characterization in the observational kinematic-age-metallicity space
• The velocity distribution of nearby stars from Hipparcos data II. The nature of the low-velocity moving groups
• The velocity distribution of nearby stars from Hipparcos data I. The significance of the moving groups
• Density estimation with heteroscedastic error
• Tracing the Hercules stream around the Galaxy

Cited by in corpus (115)

• Binary Black Hole Mergers in the first Advanced LIGO Observing Run
• Co-formation of the Galactic disc and the stellar halo
• The Thirteenth Data Release of the Sloan Digital Sky Survey: First Spectroscopic Data from the SDSS-IV Survey MApping Nearby Galaxies at Apache Point Observatory
• Gaia EDR3 view on Galactic globular clusters
• The spatial structure of mono-abundance sub-populations of the Milky Way disk
• Extracting distribution parameters from multiple uncertain observations with selection biases
• Kinematics in Young Star Clusters and Associations with Gaia DR2
• The SDSS-III Baryon Oscillation Spectroscopic Survey: Quasar Target Selection for Data Release Nine
• Proper motions and dynamics of the Milky Way globular cluster system from Gaia DR2
• APOGEE Data Releases 13 and 14: Data and Analysis
• Introduction to astroML: Machine Learning for Astrophysics
• The clustering of intermediate redshift quasars as measured by the Baryon Oscillation Spectroscopic Survey
• Star formation near the Sun is driven by expansion of the Local Bubble
• The SDSS-IV extended Baryon Oscillation Spectroscopic Survey: Quasar Target Selection
• Exploring Anticorrelations and Light Element Variations in Northern Globular Clusters Observed by the APOGEE Survey
• The total stellar halo mass of the Milky Way
• Carbon-Enhanced Metal-Poor Stars in the Inner and Outer Halo Components of the Milky Way
• The shape of the Galactic halo with $Gaia$ DR2 RR Lyrae. Anatomy of an ancient major merger
• The origin of galactic metal-rich stellar halo components with highly eccentric orbits
• Photometric redshifts and quasar probabilities from a single, data-driven generative model
• Radial Distribution of Stellar Motions in Gaia DR2
• Extended stellar systems in the solar neighborhood -- V. Discovery of coronae of nearby star clusters
• Clustering of quasars in SDSS-IV eBOSS : study of potential systematics and bias determination
• Dynamical modeling of tidal streams
• The last breath of the Sagittarius dSph
• Characterization and Photometric Performance of the Hyper Suprime-Cam Software Pipeline
• Proper motions of the Arches cluster with Keck-LGS Adaptive Optics: the first kinematic mass measurement of the Arches
• Model-independent inference on compact-binary observations
• The GALAH Survey: Chemical tagging and chrono-chemodynamics of accreted halo stars with GALAH+ DR3 and $Gaia$ eDR3
• A 14 $h^{-3}$ Gpc${}^3$ study of cosmic homogeneity using BOSS DR12 quasar sample
• Common origin for Hercules-Aquila and Virgo Clouds in Gaia DR2
• Quasar Probabilities and Redshifts from WISE mid-IR through GALEX UV Photometry
• Sloan Digital Sky Survey III Photometric Quasar Clustering: Probing the Initial Conditions of the Universe using the Largest Volume
• Exploiting the full potential of photometric quasar surveys: Optimal power spectra through blind mitigation of systematics
• Constraints on the assembly history of the Milky Way's smooth, diffuse stellar halo from the metallicity-dependent, radially-dominated velocity anisotropy profiles probed with K giants and BHB stars using LAMOST, SDSS/SEGUE, and Gaia
• Internal dynamics of the Large Magellanic Cloud from Gaia DR2
• The contribution of N-rich stars to the Galactic stellar halo using APOGEE red giants
• The Bulge Metallicity Distribution from the APOGEE Survey
• A Search for Kilonovae in the Dark Energy Survey
• The Synthetic Emission Line COSMOS catalog: H$\alpha$ and [OII] galaxy luminosity functions and counts at $0.3<z<2.5$
• The Evolution of the HeII-Ionizing Background at Redshifts 2.3<z<3.8 Inferred from a Statistical Sample of 24 HST/COS HeII Ly$\alpha$ Absorption Spectra
• Gyro-Kinematic Ages for around 30,000 Kepler Stars
• Kinematics of the Palomar 5 stellar stream from RR Lyrae stars
• Milky Way Mass with K Giants and BHB Stars Using LAMOST, SDSS/SEGUE, and Gaia: 3D Spherical Jeans Equation and Tracer Mass Estimator
• Ruprecht 147 DANCe I. Members, empirical isochrone, luminosity and mass distributions
• Likelihood Non-Gaussianity in Large-Scale Structure Analyses
• EmpiriciSN: Re-sampling Observed Supernova/Host Galaxy Populations using an XD Gaussian Mixture Model
• The Metal-Poor Stellar Halo in RAVE-TGAS and its Implications for the Velocity Distribution of Dark Matter
• Spectral and spatial analysis of the dark matter subhalo candidates among Fermi Large Area Telescope unidentified sources
• Addressing Systematics in the Traceback Age of the $\beta$ Pictoris Moving Group
• Sagittarius stream 3-d kinematics from SDSS Stripe 82
• IQ-Collaboratory 1.1: the Star-Forming Sequence of Simulated Central Galaxies
• Discovery of a disrupting open cluster far into the Milky Way halo: a recent star formation event in the leading arm of the Magellanic stream?
• Stellar ages and masses in the solar neighbourhood: Bayesian analysis using spectroscopy and Gaia DR1 parallaxes
• Membership lists for 431 open clusters in Gaia DR2 using extreme deconvolution gaussian mixture models
• Search for primordial non-Gaussianity in the quasars of SDSS-III BOSS DR9
• A Study of Quasar Selection in the Dark Energy Survey Supernova fields
• Producing a BOSS-CMASS sample with DES imaging
• Filling the gaps: Gaussian mixture models from noisy, truncated or incomplete samples
• MCR-TRGB: A Multiwavelength-Covariant, Robust Tip of the Red Giant Branch Measurement Method
• Improving \textsl{Gaia} parallax precision with a data-driven model of stars
• The effect of tides on the Sculptor dwarf spheroidal galaxy
• The seven sisters DANCe IV. Bayesian hierarchical model
• Chemically-dissected rotation curves of the Galactic Bulge from Main Sequence proper motions
• A robust estimator of mutual information for deep learning interpretability
• UOCS. III. UVIT catalogue of open clusters with machine learning based membership using \textit{Gaia} EDR3 astrometry
• Revising the Giant Planet Mass-Metallicity Relation: Deciphering the Formation Sequence of Giant Planets
• The Oceanus Moving Group: A New 500 Myr-Old Host for the Nearest Brown Dwarf
• Galactic Double Neutron Star total masses and Gaussian mixture model selection
• Two Dimensional Clustering of Gamma-Ray Bursts using durations and hardness
• Population mixtures and searches of lensed and extended quasars across photometric surveys
• A LAMOST BHB Catalog and Kinematics Therein I: Catalog and Halo Properties
• Universal properties of the high- and low-α disk: small intrinsic abundance scatter and migrating stars
• VISION - Vienna Survey in Orion II. Infrared extinction in Orion A
• Coronal loop physical parameters from the analysis of multiple observed transverse oscillations
• Ongoing hierarchical massive cluster assembly: the LISCA II structure in the Perseus complex
• Made-to-measure modeling of observed galaxy dynamics
• PopSED: Population-Level Inference for Galaxy Properties from Broadband Photometry with Neural Density Estimation
• Bayesian astrostatistics: a backward look to the future
• An extended view of the Pisces Overdensity from the SCUSS survey
• Measuring Galactic Dark Matter through Unsupervised Machine Learning
• A Unified Exploration of the Chronology of the Galaxy
• Probing the disk-corona systems and broad line regions of changing-look quasars with X-ray and optical observations
• Searching for the extra-tidal stars of globular clusters using high-dimensional analysis and a core particle spray code
• Finding $r$-II sibling stars in the Milky Way with the Greedy Optimistic Clustering algorithm
• Mapping the tilt of the Milky Way bulge velocity ellipsoids with ARGOS and $Gaia$ DR2
• Young nearby open clusters and their luminosity functions
• Kinematically Detected Halo Streams
• Paving the Way for Euclid and JWST via Optimal Selection of High-z Quasars
• The Time-Domain Spectroscopic Survey: Radial Velocity Variability in Dwarf Carbon Stars
• Tracing the Origin of Moving Groups. III. Detecting Moving Groups in The LAMOST DR7
• Blanco DECam Bulge Survey (BDBS) V. Cleaning the foreground populations from Galactic bulge colour-magnitude diagrams using Gaia EDR3
• Hierarchical probabilistic inference of the color-magnitude diagram and shrinkage of stellar distance uncertainties
• Characterization and dynamics of the peculiar stream Jhelum, A tentative role for the Sagittarius dwarf galaxy
• A Gaia DR2 search for dwarf galaxies towards Fermi-LAT sources: implications for annihilating dark matter
• The cosmic waltz of Coma Berenices and Latyshev 2 (Group X). Membership, phase-space structure, mass, and energy distributions
• An accurate strong lensing model of the Abell 2163 core
• Classification of Pulsar Glitch Amplitudes using Extreme Deconvolution
• The missing radial velocities of Gaia: blind predictions for DR3
• Classifying Exoplanets with Gaussian Mixture Model
• The Next Generation Virgo Cluster Survey. XXXIII. Stellar Population Gradients in the Virgo Cluster Core Globular Cluster System
• A 500 pc volume-limited sample of hot subluminous stars II. Atmospheric parameters, mass distribution, and kinematics
• Nonparametric Methods in Astronomy: Think, Regress, Observe -- Pick Any Three
• Gaussian Mixture Models for Blended Photometric Redshifts
• Optimal extinction measurements - I. Single-object extinction inference
• Miec: A Bayesian hierarchical model for the analysis of nearby young open clusters
• StarStream: Automatic detection algorithm for stellar streams
• An approach to robust Bayesian regression in astronomy
• Cosmology with Binary Neutron Stars: Does Mass-Redshift Correlation Matter?
• Temperate super-Earths/mini-Neptunes around M/K dwarfs Consist of 2 Populations Distinguished by Their Atmospheres
• Characterization and classification of $\gamma$-ray bursts from blazars
• Regression for Astronomical Data with Realistic Distributions, Errors and Non-linearity
• yonder: A python package for data denoising and reconstruction
• Euclid: Discovery of 31 new quasars at $6.6<z<7.8$
• Avoiding subtraction and division of stochastic signals using normalizing flows: NFdeconvolve