Preventing Posterior Collapse with delta-VAEs

Abstract

Due to the phenomenon of "posterior collapse," current latent variable generative models pose a challenging design choice that either weakens the capacity of the decoder or requires augmenting the objective so it does not only maximize the likelihood of the data. In this paper, we propose an alternative that utilizes the most powerful generative models as decoders, whilst optimising the variational lower bound all while ensuring that the latent variables preserve and encode useful information. Our proposed $\delta$-VAEs achieve this by constraining the variational family for the posterior to have a minimum distance to the prior. For sequential latent variable models, our approach resembles the classic representation learning approach of slow feature analysis. We demonstrate the efficacy of our approach at modeling text on LM1B and modeling images: learning representations, improving sample quality, and achieving state of the art log-likelihood on CIFAR-10 and ImageNet $32\times 32$.

References in corpus (18)

• Auto-Encoding Variational Bayes
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• Variational Inference with Normalizing Flows
• A Recurrent Latent Variable Model for Sequential Data
• Layer Normalization
• Stochastic Video Generation with a Learned Prior
• Variational Lossy Autoencoder
• Improving Variational Inference with Inverse Autoregressive Flow
• Hyperspherical Variational Auto-Encoders
• PixelVAE: A Latent Variable Model for Natural Images
• Stochastic Variational Video Prediction
• Uncertainty in the Variational Information Bottleneck
• VAE with a VampPrior
• Theory and Experiments on Vector Quantized Autoencoders
• Generating Sentences by Editing Prototypes
• Associative Compression Networks for Representation Learning

Cited by in corpus (10)

• Shaping Belief States with Generative Environment Models for RL
• Structured Variational Inference for Simulating Populations of Radio Galaxies
• Study of Deep Generative Models for Inorganic Chemical Compositions
• Data-Driven Offline Optimization For Architecting Hardware Accelerators
• Bridging the ELBO and MMD
• A Variational Auto-Encoder for Reservoir Monitoring
• Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders
• EXoN: EXplainable encoder Network
• Noise Contrastive Variational Autoencoders
• Inverse Graphics: Unsupervised Learning of 3D Shapes from Single Images