Rényi Divergence Variational Inference

Abstract

This paper introduces the variational Rényi bound (VR) that extends traditional variational inference to Rényi's alpha-divergences. This new family of variational methods unifies a number of existing approaches, and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood that is controlled by the value of alpha that parametrises the divergence. The reparameterization trick, Monte Carlo approximation and stochastic optimisation methods are deployed to obtain a tractable and unified framework for optimisation. We further consider negative alpha values and propose a novel variational inference method as a new special case in the proposed framework. Experiments on Bayesian neural networks and variational auto-encoders demonstrate the wide applicability of the VR bound.

NIPS 2016

Cited by in corpus (39)

• NVAE: A Deep Hierarchical Variational Autoencoder
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Generalized Variational Inference: Three arguments for deriving new Posteriors
• On the Expressiveness of Approximate Inference in Bayesian Neural Networks
• Fixing a Broken ELBO
• Variational Inference based on Robust Divergences
• Tighter Variational Bounds are Not Necessarily Better
• Approximate Inference with Amortised MCMC
• Advances in Variational Inference
• Alpha-Beta Divergence For Variational Inference
• alpha-Deep Probabilistic Inference (alpha-DPI): efficient uncertainty quantification from exoplanet astrometry to black hole feature extraction
• f-Divergence Variational Inference
• Markovian Score Climbing: Variational Inference with KL(p||q)
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• Variational f-divergence Minimization
• Reparameterization Gradient for Non-differentiable Models
• Validated Variational Inference via Practical Posterior Error Bounds
• Variational Implicit Processes
• Wasserstein Variational Inference
• Perturbative Black Box Variational Inference
• Variational approximations using Fisher divergence
• An Easy to Interpret Diagnostic for Approximate Inference: Symmetric Divergence Over Simulations
• Practical and Consistent Estimation of f-Divergences
• Hierarchical Density Order Embeddings
• GO Gradient for Expectation-Based Objectives
• Continuously tempered Hamiltonian Monte Carlo
• Stopping Criterion Design for Recursive Bayesian Classification: Analysis and Decision Geometry
• Generalized Bayesian Filtering via Sequential Monte Carlo
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• All in the Exponential Family: Bregman Duality in Thermodynamic Variational Inference
• Mixture weights optimisation for Alpha-Divergence Variational Inference
• Precision-Recall Curves Using Information Divergence Frontiers
• Learning in Variational Autoencoders with Kullback-Leibler and Renyi Integral Bounds
• Doubly Adaptive Importance Sampling
• CRAUM-Net: Contextual Recursive Attention with Uncertainty Modeling for Salient Object Detection
• PAC-Bayes Bounds on Variational Tempered Posteriors for Markov Models
• Alpha-Divergences in Variational Dropout
• Measuring Bayesian Robustness Using Rényi Divergence