Advances in Variational Inference
arXiv:1711.05597
Abstract
Many modern unsupervised or semi-supervised machine learning algorithms rely on Bayesian probabilistic models. These models are usually intractable and thus require approximate inference. Variational inference (VI) lets us approximate a high-dimensional Bayesian posterior with a simpler variational distribution by solving an optimization problem. This approach has been successfully used in various models and large-scale applications. In this review, we give an overview of recent trends in variational inference. We first introduce standard mean field variational inference, then review recent advances focusing on the following aspects: (a) scalable VI, which includes stochastic approximations, (b) generic VI, which extends the applicability of VI to a large class of otherwise intractable models, such as non-conjugate models, (c) accurate VI, which includes variational models beyond the mean field approximation or with atypical divergences, and (d) amortized VI, which implements the inference over local latent variables with inference networks. Finally, we provide a summary of promising future research directions.
References in corpus (72)
- Auto-Encoding Variational Bayes
- ADADELTA: An Adaptive Learning Rate Method
- Categorical Reparameterization with Gumbel-Softmax
- MCMC using Hamiltonian dynamics
- Trust Region Policy Optimization
- The information bottleneck method
- Loopy Belief Propagation for Approximate Inference: An Empirical Study
- Expectation Propagation for approximate Bayesian inference
- NICE: Non-linear Independent Components Estimation
- Variational Inference with Normalizing Flows
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- Determinantal point processes for machine learning
- Gaussian Processes for Big Data
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Variational Lossy Autoencoder
- Importance Weighted Autoencoders
- Edward: A library for probabilistic modeling, inference, and criticism
- Bayesian Dark Knowledge
- Frequentist Consistency of Variational Bayes
- Learning in Implicit Generative Models
- The Numerics of GANs
- Dynamic Word Embeddings
- Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
- REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
- Autoencoding Variational Inference For Topic Models
- Deep Kalman Filters
- Towards Deeper Understanding of Variational Autoencoding Models
- Streaming Variational Bayes
- Variational Bayesian Inference with Stochastic Search
- Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
- Hierarchical Implicit Models and Likelihood-Free Variational Inference
- On integral probability metrics, ϕ-divergences and binary classification
- Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling
- Learning to Draw Samples: With Application to Amortized MLE for Generative Adversarial Learning
- Fast Variational Inference in the Conjugate Exponential Family
- Stein Variational Policy Gradient
- Variational Inference using Implicit Distributions
- Rényi Divergence Variational Inference
- Variational Boosting: Iteratively Refining Posterior Approximations
- Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
- Operator Variational Inference
- Reinterpreting Importance-Weighted Autoencoders
- Structured Stochastic Variational Inference
- Boosting Variational Inference
- A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation
- Stochastic Variational Inference for Hidden Markov Models
- Nonparametric variational inference
- Reducing Reparameterization Gradient Variance
- Stochastic gradient variational Bayes for gamma approximating distributions
- Big Batch SGD: Automated Inference using Adaptive Batch Sizes
- ZhuSuan: A Library for Bayesian Deep Learning
- On Nesting Monte Carlo Estimators
- Importance Sampling for Minibatches
- Approximate Inference with Amortised MCMC
- Variational Message Passing with Structured Inference Networks
- Variational Gaussian Copula Inference
- Stein Variational Adaptive Importance Sampling
- Stochastic variational inference for large-scale discrete choice models using adaptive batch sizes
- Adversarial Message Passing For Graphical Models
- Design and Implementation of Probabilistic Programming Language Anglican
- Learning Model Reparametrizations: Implicit Variational Inference by Fitting MCMC distributions
- Overdispersed Black-Box Variational Inference
- Faster Coordinate Descent via Adaptive Importance Sampling
- A trust-region method for stochastic variational inference with applications to streaming data
- Perturbative Corrections for Approximate Inference in Gaussian Latent Variable Models
- Smoothed Gradients for Stochastic Variational Inference
- Two Methods For Wild Variational Inference
- Embarrassingly Parallel Variational Inference in Nonconjugate Models
- Structured Black Box Variational Inference for Latent Time Series Models
- Active Mini-Batch Sampling using Repulsive Point Processes
- Multimodal Prediction and Personalization of Photo Edits with Deep Generative Models
Cited by in corpus (16)
- Deep generative models in inversion: a review and development of a new approach based on a variational autoencoder
- Semi-Amortized Variational Autoencoders
- A Tutorial on Deep Latent Variable Models of Natural Language
- Stein Variational Gradient Descent Without Gradient
- Quasi-Monte Carlo Variational Inference
- Active Mini-Batch Sampling using Repulsive Point Processes
- Using Large Ensembles of Control Variates for Variational Inference
- Wasserstein Variational Inference
- The Bayesian Learning Rule
- Learning to Solve AC Optimal Power Flow by Differentiating through Holomorphic Embeddings
- On the Difficulty of Unbiased Alpha Divergence Minimization
- Scalable Approximate Inference and Some Applications
- Detect, anticipate and generate: Semi-supervised recurrent latent variable models for human activity modeling
- Multilevel Monte Carlo Variational Inference
- Approximation Based Variance Reduction for Reparameterization Gradients
- Structured Stochastic Gradient MCMC