Improved Techniques for Training Score-Based Generative Models

Abstract

Score-based generative models can produce high quality image samples comparable to GANs, without requiring adversarial optimization. However, existing training procedures are limited to images of low resolution (typically below 32x32), and can be unstable under some settings. We provide a new theoretical analysis of learning and sampling from score models in high dimensional spaces, explaining existing failure modes and motivating new solutions that generalize across datasets. To enhance stability, we also propose to maintain an exponential moving average of model weights. With these improvements, we can effortlessly scale score-based generative models to images with unprecedented resolutions ranging from 64x64 to 256x256. Our score-based models can generate high-fidelity samples that rival best-in-class GANs on various image datasets, including CelebA, FFHQ, and multiple LSUN categories.

arXiv commentNeurIPS 2020

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Conditional Image Generation with PixelCNN Decoders
• On the Convergence of Adam and Beyond
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• A Note on the Inception Score
• Assessing Generative Models via Precision and Recall
• Implicit Generation and Generalization in Energy-Based Models
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Adversarial score matching and improved sampling for image generation
• Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser

Cited by in corpus (39)

• Denoising Diffusion Probabilistic Models
• Diffusion Models Beat GANs on Image Synthesis
• Cascaded Diffusion Models for High Fidelity Image Generation
• Improved Denoising Diffusion Probabilistic Models
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Denoising Diffusion Implicit Models
• Structured Denoising Diffusion Models in Discrete State-Spaces
• How to Train Your Energy-Based Models
• NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling
• Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
• On Fast Sampling of Diffusion Probabilistic Models
• Robust Compressed Sensing MRI with Deep Generative Priors
• WaveGrad: Estimating Gradients for Waveform Generation
• Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• MatFusion: A Generative Diffusion Model for SVBRDF Capture
• Adversarial score matching and improved sampling for image generation
• Generative machine learning methods for multivariate ensemble post-processing
• Review of end-to-end speech synthesis technology based on deep learning
• Gotta Go Fast When Generating Data with Score-Based Models
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models
• Bilateral Denoising Diffusion Models
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• Symbolic Music Generation with Diffusion Models
• PROUD: PaRetO-gUided Diffusion Model for Multi-objective Generation
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Multiscale Score Matching for Out-of-Distribution Detection
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Quantum Algorithms for the Pathwise Lasso
• Improved Autoregressive Modeling with Distribution Smoothing
• Deep Generative Learning via Schrödinger Bridge
• Adaptive and Iterative Point Cloud Denoising with Score-Based Diffusion Model
• Density Ratio Estimation via Infinitesimal Classification
• Diffusion models for Handwriting Generation
• Autoencoding Under Normalization Constraints
• Probabilistic Mapping of Dark Matter by Neural Score Matching
• Diffusion Normalizing Flow