Generalized Variational Inference: Three arguments for deriving new Posteriors

Abstract

We advocate an optimization-centric view on and introduce a novel generalization of Bayesian inference. Our inspiration is the representation of Bayes' rule as infinite-dimensional optimization problem (Csiszar, 1975; Donsker and Varadhan; 1975, Zellner; 1988). First, we use it to prove an optimality result of standard Variational Inference (VI): Under the proposed view, the standard Evidence Lower Bound (ELBO) maximizing VI posterior is preferable to alternative approximations of the Bayesian posterior. Next, we argue for generalizing standard Bayesian inference. The need for this arises in situations of severe misalignment between reality and three assumptions underlying standard Bayesian inference: (1) Well-specified priors, (2) well-specified likelihoods, (3) the availability of infinite computing power. Our generalization addresses these shortcomings with three arguments and is called the Rule of Three (RoT). We derive it axiomatically and recover existing posteriors as special cases, including the Bayesian posterior and its approximation by standard VI. In contrast, approximations based on alternative ELBO-like objectives violate the axioms. Finally, we study a special case of the RoT that we call Generalized Variational Inference (GVI). GVI posteriors are a large and tractable family of belief distributions specified by three arguments: A loss, a divergence and a variational family. GVI posteriors have appealing properties, including consistency and an interpretation as approximate ELBO. The last part of the paper explores some attractive applications of GVI in popular machine learning models, including robustness and more appropriate marginals. After deriving black box inference schemes for GVI posteriors, their predictive performance is investigated on Bayesian Neural Networks and Deep Gaussian Processes, where GVI can comprehensively improve upon existing methods.

arXiv comment103 pages, 23 figures (comprehensive revision of previous version)

References in corpus (36)

• Auto-Encoding Variational Bayes
• The information bottleneck method
• Rényi Divergence and Kullback-Leibler Divergence
• Expectation Propagation for approximate Bayesian inference
• The Loss Surfaces of Multilayer Networks
• Bayesian Online Changepoint Detection
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• New approaches to Bayesian consistency
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Variational Bayesian Inference with Stochastic Search
• Sylvester Normalizing Flows for Variational Inference
• Random Feature Expansions for Deep Gaussian Processes
• Principles of Bayesian Inference using General Divergence Criteria
• Rényi Divergence Variational Inference
• Minimum scoring rule inference
• Operator Variational Inference
• The continuous Bernoulli: fixing a pervasive error in variational autoencoders
• A Primer on PAC-Bayesian Learning
• Importance Weighting and Variational Inference
• Simpler PAC-Bayesian Bounds for Hostile Data
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Variational Inference based on Robust Divergences
• Exact Gaussian Processes on a Million Data Points
• Alpha-Beta Divergence For Variational Inference
• Deep learning with differential Gaussian process flows
• Improving Explorability in Variational Inference with Annealed Variational Objectives
• Multi-resolution Multi-task Gaussian Processes
• On Using Control Variates with Stochastic Approximation for Variational Bayes and its Connection to Stochastic Linear Regression
• Variational Inference with Tail-adaptive f-Divergence
• Spatio-temporal Bayesian On-line Changepoint Detection with Model Selection
• Generic Inference in Latent Gaussian Process Models
• Good Initializations of Variational Bayes for Deep Models
• Robust Deep Gaussian Processes
• Variational approximations using Fisher divergence
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning
• Frequentist Consistency of Generalized Variational Inference

Cited by in corpus (33)

• User-friendly introduction to PAC-Bayes bounds
• The continuous Bernoulli: fixing a pervasive error in variational autoencoders
• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Bayesian Active Meta-Learning for Few Pilot Demodulation and Equalization
• Multi-resolution Multi-task Gaussian Processes
• Non-linear Multitask Learning with Deep Gaussian Processes
• f-Divergence Variational Inference
• Finite mixture models do not reliably learn the number of components
• Variational Predictive Information Bottleneck
• Bayesian Learning of Parameterised Quantum Circuits
• Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks
• Transforming Gaussian Processes With Normalizing Flows
• Generalized Posteriors in Approximate Bayesian Computation
• Robust Deep Gaussian Processes
• Parametric Gaussian Process Regressors
• VOWEL: A Local Online Learning Rule for Recurrent Networks of Probabilistic Spiking Winner-Take-All Circuits
• Variational Bayesian Decision-making for Continuous Utilities
• Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learning
• Frequentist Consistency of Generalized Variational Inference
• A Linear Bandit for Seasonal Environments
• Deep Sigma Point Processes
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• Generalized Bayesian Filtering via Sequential Monte Carlo
• Variational inference for cutting feedback in misspecified models
• Learning Partially Known Stochastic Dynamics with Empirical PAC Bayes
• On the Robustness to Misspecification of $\alpha$-Posteriors and Their Variational Approximations
• Robust Bayesian Inference for Discrete Outcomes with the Total Variation Distance
• Free Energy Minimization: A Unified Framework for Modelling, Inference, Learning,and Optimization
• Approximate Bayesian Computation with Path Signatures
• PAC${}^m$-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
• Self-explaining variational posterior distributions for Gaussian Process models
• Direct loss minimization algorithms for sparse Gaussian processes