Score-based Generative Modeling in Latent Space
arXiv:2106.05931
Abstract
Score-based generative models (SGMs) have recently demonstrated impressive results in terms of both sample quality and distribution coverage. However, they are usually applied directly in data space and often require thousands of network evaluations for sampling. Here, we propose the Latent Score-based Generative Model (LSGM), a novel approach that trains SGMs in a latent space, relying on the variational autoencoder framework. Moving from data to latent space allows us to train more expressive generative models, apply SGMs to non-continuous data, and learn smoother SGMs in a smaller space, resulting in fewer network evaluations and faster sampling. To enable training LSGMs end-to-end in a scalable and stable manner, we (i) introduce a new score-matching objective suitable to the LSGM setting, (ii) propose a novel parameterization of the score function that allows SGM to focus on the mismatch of the target distribution with respect to a simple Normal one, and (iii) analytically derive multiple techniques for variance reduction of the training objective. LSGM obtains a state-of-the-art FID score of 2.10 on CIFAR-10, outperforming all existing generative results on this dataset. On CelebA-HQ-256, LSGM is on a par with previous SGMs in sample quality while outperforming them in sampling time by two orders of magnitude. In modeling binary images, LSGM achieves state-of-the-art likelihood on the binarized OMNIGLOT dataset. Our project page and code can be found at https://nvlabs.github.io/LSGM .
arXiv commentNeurIPS 2021
References in corpus (44)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Neural Discrete Representation Learning
- Diffusion Models Beat GANs on Image Synthesis
- Semi-Supervised Learning with Deep Generative Models
- Variational Inference with Normalizing Flows
- Generative Modeling by Estimating Gradients of the Data Distribution
- Junction Tree Variational Autoencoder for Molecular Graph Generation
- Semi-Supervised Learning with Generative Adversarial Networks
- Generating Long Sequences with Sparse Transformers
- Improved Denoising Diffusion Probabilistic Models
- Making Convolutional Networks Shift-Invariant Again
- NVAE: A Deep Hierarchical Variational Autoencoder
- Variational Diffusion Models
- Variational Lossy Autoencoder
- Generalized Denoising Auto-Encoders as Generative Models
- Efficient Graph Generation with Graph Recurrent Attention Networks
- DiffWave: A Versatile Diffusion Model for Audio Synthesis
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Jukebox: A Generative Model for Music
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation
- Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
- Interpretation and Generalization of Score Matching
- Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling
- Maximum Likelihood Training of Score-Based Diffusion Models
- WaveGrad: Estimating Gradients for Waveform Generation
- A Variational Perspective on Diffusion-Based Generative Models and Score Matching
- All SMILES Variational Autoencoder
- DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
- Noise Estimation for Generative Diffusion Models
- DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors
- Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models
- Controllable and Compositional Generation with Latent-Space Energy-Based Models
- VFlow: More Expressive Generative Flows with Variational Data Augmentation
- Discriminator optimal transport
- PixelVAE++: Improved PixelVAE with Discrete Prior
- MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
- The Expressive Power of a Class of Normalizing Flow Models
- Undirected Graphical Models as Approximate Posteriors
- Symbolic Music Generation with Diffusion Models
- Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
- Diffusion Priors In Variational Autoencoders
- Efficient Learning of Generative Models via Finite-Difference Score Matching
- D2C: Diffusion-Denoising Models for Few-shot Conditional Generation