Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
arXiv:1901.05534
Abstract
The variational autoencoder (VAE) is a popular combination of deep latent variable model and accompanying variational learning technique. By using a neural inference network to approximate the model's posterior on latent variables, VAEs efficiently parameterize a lower bound on marginal data likelihood that can be optimized directly via gradient methods. In practice, however, VAE training often results in a degenerate local optimum known as "posterior collapse" where the model learns to ignore the latent variable and the approximate posterior mimics the prior. In this paper, we investigate posterior collapse from the perspective of training dynamics. We find that during the initial stages of training the inference network fails to approximate the model's true posterior, which is a moving target. As a result, the model is encouraged to ignore the latent encoding and posterior collapse occurs. Based on this observation, we propose an extremely simple modification to VAE training to reduce inference lag: depending on the model's current mutual information between latent variable and observation, we aggressively optimize the inference network before performing each model update. Despite introducing neither new model components nor significant complexity over basic VAE, our approach is able to avoid the problem of collapse that has plagued a large amount of previous work. Empirically, our approach outperforms strong autoregressive baselines on text and image benchmarks in terms of held-out likelihood, and is competitive with more complex techniques for avoiding collapse while being substantially faster.
arXiv commentICLR 2019 Conference Paper
References in corpus (5)
Cited by in corpus (57)
- Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
- CogView: Mastering Text-to-Image Generation via Transformers
- Dynamical Variational Autoencoders: A Comprehensive Review
- Implicit Generation and Generalization in Energy-Based Models
- Generative Chemical Transformer: Neural Machine Learning of Molecular Geometric Structures from Chemical Language via Attention
- Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
- Transformer-based Conditional Variational Autoencoder for Controllable Story Generation
- Simulated annealing for optimization of graphs and sequences
- Educating Text Autoencoders: Latent Representation Guidance via Denoising
- Decision-Making with Auto-Encoding Variational Bayes
- Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
- Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
- Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows
- Residual Energy-Based Models for Text
- Disentangled Recurrent Wasserstein Autoencoder
- Learning Sparse Prototypes for Text Generation
- PTP: Parallelized Tracking and Prediction with Graph Neural Networks and Diversity Sampling
- Structured Reordering for Modeling Latent Alignments in Sequence Transduction
- DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
- Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space
- Reweighted Expectation Maximization
- Counterfactual Reasoning for Fair Clinical Risk Prediction
- Learning Discrete Structured Representations by Adversarially Maximizing Mutual Information
- Dual Gaussian-based Variational Subspace Disentanglement for Visible-Infrared Person Re-Identification
- PortaSpeech: Portable and High-Quality Generative Text-to-Speech
- Mixture Content Selection for Diverse Sequence Generation
- A Critical Look at the Consistency of Causal Estimation With Deep Latent Variable Models
- Implicit Deep Latent Variable Models for Text Generation
- Do sequence-to-sequence VAEs learn global features of sentences?
- CatVRNN: Generating Category Texts via Multi-task Learning
- Failure Modes of Variational Autoencoders and Their Effects on Downstream Tasks
- A Batch Normalized Inference Network Keeps the KL Vanishing Away
- Exploring Story Generation with Multi-task Objectives in Variational Autoencoders
- APo-VAE: Text Generation in Hyperbolic Space
- Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
- Efficient Learning of Generative Models via Finite-Difference Score Matching
- Auto-Encoding Variational Bayes for Inferring Topics and Visualization
- Batch norm with entropic regularization turns deterministic autoencoders into generative models
- Improved Variational Neural Machine Translation by Promoting Mutual Information
- Variational Diffusion Autoencoders with Random Walk Sampling
- Paraphrase Generation as Unsupervised Machine Translation
- SentenceMIM: A Latent Variable Language Model
- Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
- It's LeVAsa not LevioSA! Latent Encodings for Valence-Arousal Structure Alignment
- Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation
- On Posterior Collapse and Encoder Feature Dispersion in Sequence VAEs
- Neural Gaussian Copula for Variational Autoencoder
- Topic Modeling with Wasserstein Autoencoders
- Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck
- Latte-Mix: Measuring Sentence Semantic Similarity with Latent Categorical Mixtures
- Boosting Summarization with Normalizing Flows and Aggressive Training
- A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
- Discrete Variational Attention Models for Language Generation
- Improve variational autoEncoder with auxiliary softmax multiclassifier
- Noise Contrastive Variational Autoencoders
- A Generalised Linear Model Framework for -Variational Autoencoders based on Exponential Dispersion Families
- Dense Uncertainty Estimation via an Ensemble-based Conditional Latent Variable Model