Amortized Variational Inference: A Systematic Review

Abstract

The core principle of Variational Inference (VI) is to convert the statistical inference problem of computing complex posterior probability densities into a tractable optimization problem. This property enables VI to be faster than several sampling-based techniques. However, the traditional VI algorithm is not scalable to large data sets and is unable to readily infer out-of-bounds data points without re-running the optimization process. Recent developments in the field, like stochastic-, black box-, and amortized-VI, have helped address these issues. Generative modeling tasks nowadays widely make use of amortized VI for its efficiency and scalability, as it utilizes a parameterized function to learn the approximate posterior density parameters. In this paper, we review the mathematical foundations of various VI techniques to form the basis for understanding amortized VI. Additionally, we provide an overview of the recent trends that address several issues of amortized VI, such as the amortization gap, generalization issues, inconsistent representation learning, and posterior collapse. Finally, we analyze alternate divergence measures that improve VI optimization.

Accepted for publication at the Journal of Artificial Intelligence Research (JAIR)

References in corpus (19)

• ADADELTA: An Adaptive Learning Rate Method
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Expectation Propagation for approximate Bayesian inference
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Functional Variational Bayesian Neural Networks
• Stochastic Variational Inference for Hidden Markov Models
• Reducing Reparameterization Gradient Variance
• ZhuSuan: A Library for Bayesian Deep Learning
• On the challenges of learning with inference networks on sparse, high-dimensional data
• Autoencoding Variational Autoencoder
• Celeste: Variational inference for a generative model of astronomical images
• Deep Amortized Clustering
• Consistency Regularization for Variational Auto-Encoders
• Monte Carlo Variational Auto-Encoders
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Reducing the Amortization Gap in Variational Autoencoders: A Bayesian Random Function Approach
• Stein Variational Inference for Discrete Distributions
• Variational Inference with Holder Bounds
• Generalization Gap in Amortized Inference

Cited by in corpus (1)

• Learning to solve Bayesian inverse problems: An amortized variational inference approach using Gaussian and Flow guides