Generative Modeling by Estimating Gradients of the Data Distribution

Abstract

We introduce a new generative model where samples are produced via Langevin dynamics using gradients of the data distribution estimated with score matching. Because gradients can be ill-defined and hard to estimate when the data resides on low-dimensional manifolds, we perturb the data with different levels of Gaussian noise, and jointly estimate the corresponding scores, i.e., the vector fields of gradients of the perturbed data distribution for all noise levels. For sampling, we propose an annealed Langevin dynamics where we use gradients corresponding to gradually decreasing noise levels as the sampling process gets closer to the data manifold. Our framework allows flexible model architectures, requires no sampling during training or the use of adversarial methods, and provides a learning objective that can be used for principled model comparisons. Our models produce samples comparable to GANs on MNIST, CelebA and CIFAR-10 datasets, achieving a new state-of-the-art inception score of 8.87 on CIFAR-10. Additionally, we demonstrate that our models learn effective representations via image inpainting experiments.

NeurIPS 2019 (Oral)

References in corpus (8)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• WaveNet: A Generative Model for Raw Audio
• Semi-Supervised Learning with Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• A Learned Representation For Artistic Style
• On integral probability metrics, ϕ-divergences and binary classification
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Scheduled denoising autoencoders

Cited by in corpus (151)

• Denoising Diffusion Probabilistic Models
• On the Opportunities and Risks of Foundation Models
• Diffusion Models Beat GANs on Image Synthesis
• Diffusion Models in Vision: A Survey
• Cascaded Diffusion Models for High Fidelity Image Generation
• A Comprehensive Survey on Deep Graph Representation Learning
• Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach
• Variational Diffusion Models
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• Adversarial Text-to-Image Synthesis: A Review
• Score-Based Point Cloud Denoising
• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• VideoGPT: Video Generation using VQ-VAE and Transformers
• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• Crystal Diffusion Variational Autoencoder for Periodic Material Generation
• Denoising Diffusion Implicit Models
• SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models
• Vector-quantized Image Modeling with Improved VQGAN
• DeScoD-ECG: Deep Score-Based Diffusion Model for ECG Baseline Wander and Noise Removal
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Unsupervised Deep Learning Methods for Biological Image Reconstruction and Enhancement
• Data synthesis and adversarial networks: A review and meta-analysis in cancer imaging
• Structured Denoising Diffusion Models in Discrete State-Spaces
• A Diffusion model for POI recommendation
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
• NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling
• Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
• IterativePFN: True Iterative Point Cloud Filtering
• On Fast Sampling of Diffusion Probabilistic Models
• NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates
• Your GAN is Secretly an Energy-based Model and You Should use Discriminator Driven Latent Sampling
• PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics
• medigan: a Python library of pretrained generative models for medical image synthesis
• Maximum Likelihood Training of Score-Based Diffusion Models
• Robust Compressed Sensing MRI with Deep Generative Priors
• WaveGrad: Estimating Gradients for Waveform Generation
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
• Noise Estimation for Generative Diffusion Models
• CaloMan: Fast generation of calorimeter showers with density estimation on learned manifolds
• Con-CDVAE: A method for the conditional generation of crystal structures
• Fast Graph Generation via Spectral Diffusion
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Generative diffusion in very large dimensions
• Generating Comprehensive Lithium Battery Charging Data with Generative AI
• Adversarial score matching and improved sampling for image generation
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Improved Contrastive Divergence Training of Energy Based Models
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• PatchNR: Learning from Very Few Images by Patch Normalizing Flow Regularization
• Controllable Data Generation by Deep Learning: A Review
• Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
• Ensemble flow reconstruction in the atmospheric boundary layer from spatially limited measurements through latent diffusion models
• Permutation Invariant Graph Generation via Score-Based Generative Modeling
• Wavelet Transform-assisted Adaptive Generative Modeling for Colorization
• Conditional score-based diffusion models for solving inverse problems in mechanics
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• DeTiME: Diffusion-Enhanced Topic Modeling using Encoder-decoder based LLM
• Score-Based Generative Models for PET Image Reconstruction
• A Gentle Introduction and Tutorial on Deep Generative Models in Transportation Research
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
• Non Gaussian Denoising Diffusion Models
• Deep Generative Imputation Model for Missing Not At Random Data
• Iterative $\alpha$-(de)Blending: a Minimalist Deterministic Diffusion Model
• Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
• Solving Inverse Problems with Score-Based Generative Priors learned from Noisy Data
• A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
• Treatment-aware Diffusion Probabilistic Model for Longitudinal MRI Generation and Diffuse Glioma Growth Prediction
• Score-based Generative Modeling in Latent Space
• Geometric-Facilitated Denoising Diffusion Model for 3D Molecule Generation
• Hyperparameter Optimization of Generative Adversarial Network Models for High-Energy Physics Simulations
• Highly Accurate Real-space Electron Densities with Neural Networks
• Source Separation with Deep Generative Priors
• Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
• Symbolic Music Generation with Diffusion Models
• Riemannian Denoising Diffusion Probabilistic Models
• VoiceGrad: Non-Parallel Any-to-Many Voice Conversion with Annealed Langevin Dynamics
• A Flow-based Truncated Denoising Diffusion Model for Super-resolution Magnetic Resonance Spectroscopic Imaging
• Conditional Image Generation with Score-Based Diffusion Models
• PROUD: PaRetO-gUided Diffusion Model for Multi-objective Generation
• Unraveling the Potential of Diffusion Models in Small Molecule Generation
• Global Search for Optimal Low Thrust Spacecraft Trajectories using Diffusion Models and the Indirect Method
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions
• Oblique Decision Trees from Derivatives of ReLU Networks
• Rethinking the Role of Gradient-Based Attribution Methods for Model Interpretability
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Multiscale Score Matching for Out-of-Distribution Detection
• Neural SDEs as Infinite-Dimensional GANs
• No MCMC for me: Amortized sampling for fast and stable training of energy-based models
• Score-matching neural networks for improved multi-band source separation
• Score-Based Data Generation for EEG Spatial Covariance Matrices: Towards Boosting BCI Performance
• SNIPS: Solving Noisy Inverse Problems Stochastically
• Anytime Sampling for Autoregressive Models via Ordered Autoencoding
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Normalizing Flows Across Dimensions
• Chunked Autoregressive GAN for Conditional Waveform Synthesis
• On Energy-Based Models with Overparametrized Shallow Neural Networks
• Multi-Resolution Diffusion for Privacy-Sensitive Recommender Systems
• Blindness of score-based methods to isolated components and mixing proportions
• Telescoping Density-Ratio Estimation
• Bidirectional Generative Modeling Using Adversarial Gradient Estimation
• Generative Modeling with Optimal Transport Maps
• Fast Convergence for Langevin Diffusion with Manifold Structure
• Taming GANs with Lookahead-Minmax
• Flow Contrastive Estimation of Energy-Based Models
• EdiTTS: Score-based Editing for Controllable Text-to-Speech
• Dreaming of Electrical Waves: Generative Modeling of Cardiac Excitation Waves using Diffusion Models
• A Mathematical Framework for Learning Probability Distributions
• Refining Deep Generative Models via Discriminator Gradient Flow
• Density Ratio Estimation via Infinitesimal Classification
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• Zero Shot Molecular Generation via Similarity Kernels
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• Novel Concept-Oriented Synthetic Data approach for Training Generative AI-Driven Crystal Grain Analysis Using Diffusion Model
• Parallel and Flexible Sampling from Autoregressive Models via Langevin Dynamics
• Lessons Learned from the Training of GANs on Artificial Datasets
• Autoregressive Score Matching
• Tractable Density Estimation on Learned Manifolds with Conformal Embedding Flows
• Homotopic Gradients of Generative Density Priors for MR Image Reconstruction
• Language Guided Adversarial Purification
• Gradient Origin Networks
• A multiscale and multicriteria Generative Adversarial Network to synthesize 1-dimensional turbulent fields
• Turbulent Injection assisted by Diffusion Models for Scale Resolving Simulations
• Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities
• U-Turn Diffusion
• Generative-Model-Based Fully 3D PET Image Reconstruction by Conditional Diffusion Sampling
• Active Slices for Sliced Stein Discrepancy
• Towards Robust Active Feature Acquisition
• Autoregressive Diffusion Models
• Learning Markov Random Fields for Combinatorial Structures via Sampling through Lovász Local Lemma
• Distribution Preserving Source Separation With Time Frequency Predictive Models
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
• Maximum entropy exploration in contextual bandits with neural networks and energy based models
• Energy-Based Models for Predicting Mutational Effects on Proteins
• IgCONDA-PET: Weakly-Supervised PET Anomaly Detection using Implicitly-Guided Attention-Conditional Counterfactual Diffusion Modeling -- a Multi-Center, Multi-Cancer, and Multi-Tracer Study
• Stochastic Super-Resolution For Gaussian Textures
• Decoupling Global and Local Representations via Invertible Generative Flows
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs
• Imitation with Neural Density Models
• Bridging the Gap Between $f$-GANs and Wasserstein GANs
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• EBMs Trained with Maximum Likelihood are Generator Models Trained with a Self-adverserial Loss
• Injecting Hierarchy with U-Net Transformers
• NP-DRAW: A Non-Parametric Structured Latent Variable Model for Image Generation