Variational Tempering

Abstract

Variational inference (VI) combined with data subsampling enables approximate posterior inference over large data sets, but suffers from poor local optima. We first formulate a deterministic annealing approach for the generic class of conditionally conjugate exponential family models. This approach uses a decreasing temperature parameter which deterministically deforms the objective during the course of the optimization. A well-known drawback to this annealing approach is the choice of the cooling schedule. We therefore introduce variational tempering, a variational algorithm that introduces a temperature latent variable to the model. In contrast to related work in the Markov chain Monte Carlo literature, this algorithm results in adaptive annealing schedules. Lastly, we develop local variational tempering, which assigns a latent temperature to each data point; this allows for dynamic annealing that varies across data. Compared to the traditional VI, all proposed approaches find improved predictive likelihoods on held-out data.

published version, 8 pages, 4 figures

Cited by in corpus (11)

• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples
• How Good is the Bayes Posterior in Deep Neural Networks Really?
• An introduction to variational inference in Geophysical inverse problems
• Deep Generative Video Compression
• Learning of state-space models with highly informative observations: a tempered Sequential Monte Carlo solution
• Determinantal Point Processes for Mini-Batch Diversification
• Annealed Stein Variational Gradient Descent
• On Stein Variational Neural Network Ensembles
• Self-Paced Deep Reinforcement Learning
• The Variational InfoMax Learning Objective
• A Probabilistic Interpretation of Self-Paced Learning with Applications to Reinforcement Learning