Avoiding Latent Variable Collapse With Generative Skip Models

Abstract

Variational autoencoders learn distributions of high-dimensional data. They model data with a deep latent-variable model and then fit the model by maximizing a lower bound of the log marginal likelihood. VAEs can capture complex distributions, but they can also suffer from an issue known as "latent variable collapse," especially if the likelihood model is powerful. Specifically, the lower bound involves an approximate posterior of the latent variables; this posterior "collapses" when it is set equal to the prior, i.e., when the approximate posterior is independent of the data. While VAEs learn good generative models, latent variable collapse prevents them from learning useful representations. In this paper, we propose a simple new way to avoid latent variable collapse by including skip connections in our generative model; these connections enforce strong links between the latent variables and the likelihood function. We study generative skip models both theoretically and empirically. Theoretically, we prove that skip models increase the mutual information between the observations and the inferred latent variables. Empirically, we study images (MNIST and Omniglot) and text (Yahoo). Compared to existing VAE architectures, we show that generative skip models maintain similar predictive performance but lead to less collapse and provide more meaningful representations of the data.

arXiv commentIn the Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS 2019), Naha, Okinawa, Japan. PMLR: Volume 89. An earlier version of this paper was presented at the Workshop on Theoretical Foundations and Applications of Deep Generative Models, ICML, 2018

References in corpus (13)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Conditional Image Generation with PixelCNN Decoders
• Ladder Variational Autoencoders
• TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
• Towards Deeper Understanding of Variational Autoencoding Models
• Variational Autoencoders for Collaborative Filtering
• VAE with a VampPrior
• Tackling Over-pruning in Variational Autoencoders
• Semi-Amortized Variational Autoencoders
• Generating Sentences by Editing Prototypes
• Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation

Cited by in corpus (20)

• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
• Prescribed Generative Adversarial Networks
• A Tutorial on Deep Latent Variable Models of Natural Language
• The Usual Suspects? Reassessing Blame for VAE Posterior Collapse
• Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• Sparsity in Variational Autoencoders
• Autoencoder Image Interpolation by Shaping the Latent Space
• Hierarchical VAEs Know What They Don't Know
• The Posterior Predictive Null
• Curriculum By Smoothing
• Practical and Consistent Estimation of f-Divergences
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Bridging the ELBO and MMD
• Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders
• Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images
• Noise Contrastive Variational Autoencoders
• Enhancing audio quality for expressive Neural Text-to-Speech