Nearly Consistent Finite Particle Estimates in Streaming Importance Sampling
arXiv:1909.10279
Abstract
In Bayesian inference, we seek to compute information about random variables such as moments or quantiles on the basis of {available data} and prior information. When the distribution of random variables is {intractable}, Monte Carlo (MC) sampling is usually required. {Importance sampling is a standard MC tool that approximates this unavailable distribution with a set of weighted samples.} This procedure is asymptotically consistent as the number of MC samples (particles) go to infinity. However, retaining infinitely many particles is intractable. Thus, we propose a way to only keep a \emph{finite representative subset} of particles and their augmented importance weights that is \emph{nearly consistent}. To do so in {an online manner}, we (1) embed the posterior density estimate in a reproducing kernel Hilbert space (RKHS) through its kernel mean embedding; and (2) sequentially project this RKHS element onto a lower-dimensional subspace in RKHS using the maximum mean discrepancy, an integral probability metric. Theoretically, we establish that this scheme results in a bias determined by a compression parameter, which yields a tunable tradeoff between consistency and memory. In experiments, we observe the compressed estimates achieve comparable performance to the dense ones with substantial reductions in representational complexity.
References in corpus (14)
- Variational Inference: A Review for Statisticians
- Expectation Propagation for approximate Bayesian inference
- Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems
- Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
- Fighting Sample Degeneracy and Impoverishment in Particle Filters: A Review of Intelligent Approaches
- Hilbert space embeddings and metrics on probability measures
- Sharp failure rates for the bootstrap particle filter in high dimensions
- Improving Population Monte Carlo: Alternative Weighting and Resampling Schemes
- Adapting the Number of Particles in Sequential Monte Carlo Methods through an Online Scheme for Convergence Assessment
- Rethinking the Effective Sample Size
- Automated Scalable Bayesian Inference via Hilbert Coresets
- Compressed Monte Carlo with application in particle filtering
- Sequential Kernel Herding: Frank-Wolfe Optimization for Particle Filtering
- Metropolis Sampling