Alpha-Beta Divergence For Variational Inference

Abstract

This paper introduces a variational approximation framework using direct optimization of what is known as the {\it scale invariant Alpha-Beta divergence} (sAB divergence). This new objective encompasses most variational objectives that use the Kullback-Leibler, the R{é}nyi or the gamma divergences. It also gives access to objective functions never exploited before in the context of variational inference. This is achieved via two easy to interpret control parameters, which allow for a smooth interpolation over the divergence space while trading-off properties such as mass-covering of a target distribution and robustness to outliers in the data. Furthermore, the sAB variational objective can be optimized directly by repurposing existing methods for Monte Carlo computation of complex variational objectives, leading to estimates of the divergence instead of variational lower bounds. We show the advantages of this objective on Bayesian models for regression problems.

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Rényi Divergence and Kullback-Leibler Divergence
• Edward: A library for probabilistic modeling, inference, and criticism
• Rényi Divergence Variational Inference
• Variational Inference based on Robust Divergences

Cited by in corpus (6)

• Generalized Variational Inference: Three arguments for deriving new Posteriors
• f-Divergence Variational Inference
• On the Difficulty of Unbiased Alpha Divergence Minimization
• Mixability of Integral Losses: a Key to Efficient Online Aggregation of Functional and Probabilistic Forecasts
• Variational Inference MPC using Tsallis Divergence
• The Renyi Gaussian Process: Towards Improved Generalization