Amortized Inference Regularization

Abstract

The variational autoencoder (VAE) is a popular model for density estimation and representation learning. Canonically, the variational principle suggests to prefer an expressive inference model so that the variational approximation is accurate. However, it is often overlooked that an overly-expressive inference model can be detrimental to the test set performance of both the amortized posterior approximator and, more importantly, the generative density estimator. In this paper, we leverage the fact that VAEs rely on amortized inference and propose techniques for amortized inference regularization (AIR) that control the smoothness of the inference model. We demonstrate that, by applying AIR, it is possible to improve VAE generalization on both inference and generative performance. Our paper challenges the belief that amortized inference is simply a mechanism for approximating maximum likelihood training and illustrates that regularization of the amortization family provides a new direction for understanding and improving generalization in VAEs.

arXiv commentNeurIPS 2018

Cited by in corpus (19)

• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance
• Calibration of Deep Probabilistic Models with Decoupled Bayesian Neural Networks
• Probabilistic Model-Agnostic Meta-Learning
• Multi-class Gaussian Process Classification with Noisy Inputs
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• Meta-Amortized Variational Inference and Learning
• Variational Bayesian Context-aware Representation for Grocery Recommendation
• Multi-objects Generation with Amortized Structural Regularization
• Towards Amortized Ranking-Critical Training for Collaborative Filtering
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• Rate-Regularization and Generalization in VAEs
• "Best-of-Many-Samples" Distribution Matching
• Input Dependent Sparse Gaussian Processes
• Variational Autoencoders: A Harmonic Perspective
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Quantization-Based Regularization for Autoencoders
• Modeling Graph Node Correlations with Neighbor Mixture Models