Amortized Variational Inference: A Systematic Review

Abstract

The core principle of Variational Inference (VI) is to convert the statistical inference problem of computing complex posterior probability densities into a tractable optimization problem. This property enables VI to be faster than several sampling-based techniques. However, the traditional VI algorithm is not scalable to large data sets and is unable to readily infer out-of-bounds data points without re-running the optimization process. Recent developments in the field, like stochastic-, black box-, and amortized-VI, have helped address these issues. Generative modeling tasks nowadays widely make use of amortized VI for its efficiency and scalability, as it utilizes a parameterized function to learn the approximate posterior density parameters. In this paper, we review the mathematical foundations of various VI techniques to form the basis for understanding amortized VI. Additionally, we provide an overview of the recent trends that address several issues of amortized VI, such as the amortization gap, generalization issues, inconsistent representation learning, and posterior collapse. Finally, we analyze alternate divergence measures that improve VI optimization.

arXiv commentAccepted for publication at the Journal of Artificial Intelligence Research (JAIR)

References in corpus (47)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• ADADELTA: An Adaptive Learning Rate Method
• Categorical Reparameterization with Gumbel-Softmax
• MCMC using Hamiltonian dynamics
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Expectation Propagation for approximate Bayesian inference
• Variational Inference with Normalizing Flows
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Autoencoding beyond pixels using a learned similarity metric
• NVAE: A Deep Hierarchical Variational Autoencoder
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• New insights and perspectives on the natural gradient method
• Understanding disentangling in $\beta$-VAE
• Importance Weighted Autoencoders
• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
• Functional Variational Bayesian Neural Networks
• Improved Variational Autoencoders for Text Modeling using Dilated Convolutions
• HypoSVI: Hypocenter inversion with Stein variational inference and Physics Informed Neural Networks
• Monte Carlo Gradient Estimation in Machine Learning
• Inference Suboptimality in Variational Autoencoders
• Getting started in probabilistic graphical models
• Variational Autoencoders for Collaborative Filtering
• Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam
• Metrics for Deep Generative Models
• Stochastic Variational Inference for Hidden Markov Models
• Reducing Reparameterization Gradient Variance
• Stochastic gradient variational Bayes for gamma approximating distributions
• ZhuSuan: A Library for Bayesian Deep Learning
• On the challenges of learning with inference networks on sparse, high-dimensional data
• Tighter Variational Bounds are Not Necessarily Better
• Autoencoding Variational Autoencoder
• Celeste: Variational inference for a generative model of astronomical images
• Deep Amortized Clustering
• Consistency Regularization for Variational Auto-Encoders
• Variational Inference: A Unified Framework of Generative Models and Some Revelations
• Monte Carlo Variational Auto-Encoders
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Generalization Gap in Amortized Inference
• Stein Variational Inference for Discrete Distributions
• Variational Inference with Holder Bounds
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach

Cited by in corpus (1)

• Learning to solve Bayesian inverse problems: An amortized variational inference approach using Gaussian and Flow guides