VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
arXiv:2010.00654
Abstract
Energy-based models (EBMs) have recently been successful in representing complex distributions of small images. However, sampling from them requires expensive Markov chain Monte Carlo (MCMC) iterations that mix slowly in high dimensional pixel space. Unlike EBMs, variational autoencoders (VAEs) generate samples quickly and are equipped with a latent space that enables fast traversal of the data manifold. However, VAEs tend to assign high probability density to regions in data space outside the actual data distribution and often fail at generating sharp images. In this paper, we propose VAEBM, a symbiotic composition of a VAE and an EBM that offers the best of both worlds. VAEBM captures the overall mode structure of the data distribution using a state-of-the-art VAE and it relies on its EBM component to explicitly exclude non-data-like regions from the model and refine the image samples. Moreover, the VAE component in VAEBM allows us to speed up MCMC updates by reparameterizing them in the VAE's latent space. Our experimental results show that VAEBM outperforms state-of-the-art VAEs and EBMs in generative quality on several benchmark image datasets by a large margin. It can generate high-quality images as large as 256256 pixels with short MCMC chains. We also demonstrate that VAEBM provides complete mode coverage and performs well in out-of-distribution detection. The source code is available at https://github.com/NVlabs/VAEBM
arXiv commentICLR 2021 (spotlight)
References in corpus (30)
- Auto-Encoding Variational Bayes
- Denoising Diffusion Probabilistic Models
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Spectral Normalization for Generative Adversarial Networks
- Semi-Supervised Learning with Deep Generative Models
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Variational Inference with Normalizing Flows
- Generative Modeling by Estimating Gradients of the Data Distribution
- Energy-based Generative Adversarial Network
- Density estimation using Real NVP
- VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
- A Note on the Inception Score
- Equivariant flow-based sampling for lattice gauge theory
- A Theory of Generative ConvNet
- Improved Techniques for Training Score-Based Generative Models
- Jukebox: A Generative Model for Music
- NeuTra-lizing Bad Geometry in Hamiltonian Monte Carlo Using Neural Transport
- WaveFlow: A Compact Flow-based Model for Raw Audio
- Prescribed Generative Adversarial Networks
- DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
- Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
- Autoregressive Quantile Networks for Generative Modeling
- DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
- Hybrid Discriminative-Generative Training via Contrastive Learning
- Quality Aware Generative Adversarial Networks
- Undirected Graphical Models as Approximate Posteriors
- The Expressive Power of a Class of Normalizing Flow Models
- Discriminator Contrastive Divergence: Semi-Amortized Generative Modeling by Exploring Energy of the Discriminator
- Exponential Tilting of Generative Models: Improving Sample Quality by Training and Sampling from Latent Energy