A Generalization Bound for Online Variational Inference

Abstract

Bayesian inference provides an attractive online-learning framework to analyze sequential data, and offers generalization guarantees which hold even with model mismatch and adversaries. Unfortunately, exact Bayesian inference is rarely feasible in practice and approximation methods are usually employed, but do such methods preserve the generalization properties of Bayesian inference ? In this paper, we show that this is indeed the case for some variational inference (VI) algorithms. We consider a few existing online, tempered VI algorithms, as well as a new algorithm, and derive their generalization bounds. Our theoretical result relies on the convexity of the variational objective, but we argue that the result should hold more generally and present empirical evidence in support of this. Our work in this paper presents theoretical justifications in favor of online algorithms relying on approximate Bayesian methods.

Published in the proceedings of ACML 2019

References in corpus (10)

• Expectation Propagation for approximate Bayesian inference
• Fast learning rates in statistical inference through aggregation
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• A Primer on PAC-Bayesian Learning
• Consistency of Variational Bayes Inference for Estimation and Model Selection in Mixtures
• Consistency of ELBO maximization for model selection
• Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis
• Bayesian fractional posteriors
• Provable Smoothness Guarantees for Black-Box Variational Inference
• Asymptotic Consistency of $\alpha -$Rényi-Approximate Posteriors

Cited by in corpus (5)

• MMD-Bayes: Robust Bayesian Estimation via Maximum Mean Discrepancy
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• Meta-strategy for Learning Tuning Parameters with Guarantees
• Practical calibration of the temperature parameter in Gibbs posteriors