Semi-Implicit Variational Inference

Abstract

Semi-implicit variational inference (SIVI) is introduced to expand the commonly used analytic variational distribution family, by mixing the variational parameter with a flexible distribution. This mixing distribution can assume any density function, explicit or not, as long as independent random samples can be generated via reparameterization. Not only does SIVI expand the variational family to incorporate highly flexible variational distributions, including implicit ones that have no analytic density functions, but also sandwiches the evidence lower bound (ELBO) between a lower bound and an upper bound, and further derives an asymptotically exact surrogate ELBO that is amenable to optimization via stochastic gradient ascent. With a substantially expanded variational family and a novel optimization algorithm, SIVI is shown to closely match the accuracy of MCMC in inferring the posterior in a variety of Bayesian inference tasks.

arXiv commentICML 2018

References in corpus (22)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• DRAW: A Recurrent Neural Network For Image Generation
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Variational Relevance Vector Machines
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Auxiliary Deep Generative Models
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Learning in Implicit Generative Models
• Variational Bayesian Inference with Stochastic Search
• Hierarchical Implicit Models and Likelihood-Free Variational Inference
• Lognormal and Gamma Mixed Negative Binomial Regression
• Denoising Criterion for Variational Auto-Encoding Framework
• Variational Inference using Implicit Distributions
• Gradient Estimators for Implicit Models
• Boosting Variational Inference
• Copula variational inference
• The Generalized Reparameterization Gradient
• Tighter Variational Bounds are Not Necessarily Better
• Variational Gaussian Copula Inference
• Softplus Regressions and Convex Polytopes
• Kernel Implicit Variational Inference

Cited by in corpus (12)

• Learning on Attribute-Missing Graphs
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Infinite-dimensional gradient-based descent for alpha-divergence minimisation
• Recursive Inference for Variational Autoencoders
• AMAGOLD: Amortized Metropolis Adjustment for Efficient Stochastic Gradient MCMC
• Doubly Semi-Implicit Variational Inference
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Variationally Inferred Sampling Through a Refined Bound for Probabilistic Programs
• Contributions to Large Scale Bayesian Inference and Adversarial Machine Learning
• Statistical Guarantees for Transformation Based Models with Applications to Implicit Variational Inference
• Semi-Implicit Stochastic Recurrent Neural Networks
• Uncertainty in Neural Processes