A Contrastive Learning Approach for Training Variational Autoencoder Priors
arXiv:2010.02917
Abstract
Variational autoencoders (VAEs) are one of the powerful likelihood-based generative models with applications in many domains. However, they struggle to generate high-quality images, especially when samples are obtained from the prior without any tempering. One explanation for VAEs' poor generative quality is the prior hole problem: the prior distribution fails to match the aggregate approximate posterior. Due to this mismatch, there exist areas in the latent space with high density under the prior that do not correspond to any encoded image. Samples from those areas are decoded to corrupted images. To tackle this issue, we propose an energy-based prior defined by the product of a base prior distribution and a reweighting factor, designed to bring the base closer to the aggregate posterior. We train the reweighting factor by noise contrastive estimation, and we generalize it to hierarchical VAEs with many latent variable groups. Our experiments confirm that the proposed noise contrastive priors improve the generative performance of state-of-the-art VAEs by a large margin on the MNIST, CIFAR-10, CelebA 64, and CelebA HQ 256 datasets. Our method is simple and can be applied to a wide variety of VAEs to improve the expressivity of their prior distribution.
arXiv commentAccepted to NeurIPS 2021
References in corpus (33)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Denoising Diffusion Probabilistic Models
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Neural Discrete Representation Learning
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Pixel Recurrent Neural Networks
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Semi-Supervised Learning with Deep Generative Models
- Training Generative Adversarial Networks with Limited Data
- Searching for Activation Functions
- Variational Lossy Autoencoder
- Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Jukebox: A Generative Model for Music
- SOM-VAE: Interpretable Discrete Representation Learning on Time Series
- Bias Correction of Learned Generative Models using Likelihood-Free Importance Weighting
- WaveFlow: A Compact Flow-based Model for Raw Audio
- Distribution Matching in Variational Inference
- Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
- Discrete Variational Autoencoders
- DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
- DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
- Discriminator Rejection Sampling
- Autoregressive Energy Machines
- Quality Aware Generative Adversarial Networks
- Perceptual Generative Autoencoders
- Undirected Graphical Models as Approximate Posteriors
- On the Necessity and Effectiveness of Learning the Prior of Variational Auto-Encoder
- Learning Latent Space Energy-Based Prior Model