Score-based Generative Modeling in Latent Space
arXiv:2106.05931
Abstract
Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our project page and code can be found at https://nvlabs.github.io/LSGM .
NeurIPS 2021
References in corpus (26)
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Diffusion Models Beat GANs on Image Synthesis
- Semi-Supervised Learning with Deep Generative Models
- Generating Long Sequences with Sparse Transformers
- Improved Denoising Diffusion Probabilistic Models
- Making Convolutional Networks Shift-Invariant Again
- Variational Diffusion Models
- Variational Lossy Autoencoder
- DiffWave: A Versatile Diffusion Model for Audio Synthesis
- Jukebox: A Generative Model for Music
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation
- Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
- Interpretation and Generalization of Score Matching
- Maximum Likelihood Training of Score-Based Diffusion Models
- WaveGrad: Estimating Gradients for Waveform Generation
- All SMILES Variational Autoencoder
- A Variational Perspective on Diffusion-Based Generative Models and Score Matching
- Noise Estimation for Generative Diffusion Models
- Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
- Controllable and Compositional Generation with Latent-Space Energy-Based Models
- MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
- The Expressive Power of a Class of Normalizing Flow Models
- Symbolic Music Generation with Diffusion Models
- Diffusion Priors In Variational Autoencoders
- Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
- D2C: Diffusion-Denoising Models for Few-shot Conditional Generation