Automated Scalable Bayesian Inference via Hilbert Coresets

Abstract

The automation of posterior inference in Bayesian data analysis has enabled experts and nonexperts alike to use more sophisticated models, engage in faster exploratory modeling and analysis, and ensure experimental reproducibility. However, standard automated posterior inference algorithms are not tractable at the scale of massive modern datasets, and modifications to make them so are typically model-specific, require expert tuning, and can break theoretical guarantees on inferential quality. Building on the Bayesian coresets framework, this work instead takes advantage of data redundancy to shrink the dataset itself as a preprocessing step, providing fully-automated, scalable Bayesian inference with theoretical guarantees. We begin with an intuitive reformulation of Bayesian coreset construction as sparse vector sum approximation, and demonstrate that its automation and performance-based shortcomings arise from the use of the supremum norm. To address these shortcomings we develop Hilbert coresets, i.e., Bayesian coresets constructed under a norm induced by an inner-product on the log-likelihood function space. We propose two Hilbert coreset construction algorithms---one based on importance sampling, and one based on the Frank-Wolfe algorithm---along with theoretical guarantees on approximation quality as a function of coreset size. Since the exact computation of the proposed inner-products is model-specific, we automate the construction with a random finite-dimensional projection of the log-likelihood functions. The resulting automated coreset construction algorithm is simple to implement, and experiments on a variety of models with real and synthetic datasets show that it provides high-quality posterior approximations and a significant reduction in the computational cost of inference.

Cited by in corpus (28)

• Selection via Proxy: Efficient Data Selection for Deep Learning
• Coresets via Bilevel Optimization for Continual Learning and Streaming
• Bayesian Batch Active Learning as Sparse Subset Approximation
• Online Coreset Selection for Rehearsal-based Continual Learning
• Decentralized Stochastic Gradient Langevin Dynamics and Hamiltonian Monte Carlo
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• Continuous Regularized Wasserstein Barycenters
• Bayesian Coreset Construction via Greedy Iterative Geodesic Ascent
• Approximations of Geometrically Ergodic Reversible Markov Chains
• Small quantum computers and large classical data sets
• Active learning for interactive satellite image change detection
• Spectral Subsampling MCMC for Stationary Time Series
• Sparse Variational Inference: Bayesian Coresets from Scratch
• Wasserstein Measure Coresets
• Nearly Consistent Finite Particle Estimates in Streaming Importance Sampling
• Online Sampling from Log-Concave Distributions
• Scalable Gaussian Process Inference with Finite-data Mean and Variance Guarantees
• Bayesian Active Learning by Disagreements: A Geometric Perspective
• Greedy Frank-Wolfe Algorithm for Exemplar Selection
• Data Summarization via Bilevel Optimization
• Efficient posterior sampling for high-dimensional imbalanced logistic regression
• Bayesian Coresets: Revisiting the Nonconvex Optimization Perspective
• One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning
• Targeted stochastic gradient Markov chain Monte Carlo for hidden Markov models with rare latent states
• Consensus Monte Carlo for Random Subsets using Shared Anchors
• Risk Bounds for Learning via Hilbert Coresets
• Face-NMS: A Core-set Selection Approach for Efficient Face Recognition
• Model-specific Data Subsampling with Influence Functions