Score-Based Generative Modeling through Stochastic Differential Equations

Abstract

Creating noise from data is easy; creating data from noise is generative modeling. We present a stochastic differential equation (SDE) that smoothly transforms a complex data distribution to a known prior distribution by slowly injecting noise, and a corresponding reverse-time SDE that transforms the prior distribution back into the data distribution by slowly removing the noise. Crucially, the reverse-time SDE depends only on the time-dependent gradient field (\aka, score) of the perturbed data distribution. By leveraging advances in score-based generative modeling, we can accurately estimate these scores with neural networks, and use numerical SDE solvers to generate samples. We show that this framework encapsulates previous approaches in score-based generative modeling and diffusion probabilistic modeling, allowing for new sampling procedures and new modeling capabilities. In particular, we introduce a predictor-corrector framework to correct errors in the evolution of the discretized reverse-time SDE. We also derive an equivalent neural ODE that samples from the same distribution as the SDE, but additionally enables exact likelihood computation, and improved sampling efficiency. In addition, we provide a new way to solve inverse problems with score-based models, as demonstrated with experiments on class-conditional generation, image inpainting, and colorization. Combined with multiple architectural improvements, we achieve record-breaking performance for unconditional image generation on CIFAR-10 with an Inception score of 9.89 and FID of 2.20, a competitive likelihood of 2.99 bits/dim, and demonstrate high fidelity generation of 1024 x 1024 images for the first time from a score-based generative model.

ICLR 2021 (Oral)

References in corpus (10)

• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Wide Residual Networks
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Density estimation using Real NVP
• Making Convolutional Networks Shift-Invariant Again
• Residual Flows for Invertible Generative Modeling
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimation
• Adversarial score matching and improved sampling for image generation
• On Linear Identifiability of Learned Representations

Cited by in corpus (116)

• Diffusion Models Beat GANs on Image Synthesis
• Diffusion Models in Vision: A Survey
• Towards Small Object Editing: A Benchmark Dataset and A Training-Free Approach
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Adaptive Diffusion Priors for Accelerated MRI Reconstruction
• Adversarial Text-to-Image Synthesis: A Review
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
• Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models
• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• At the Dawn of Generative AI Era: A Tutorial-cum-Survey on New Frontiers in 6G Wireless Intelligence
• InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models
• Zero-shot Medical Image Translation via Frequency-Guided Diffusion Models
• DeScoD-ECG: Deep Score-Based Diffusion Model for ECG Baseline Wander and Noise Removal
• Unsupervised Deep Learning Methods for Biological Image Reconstruction and Enhancement
• Structured Denoising Diffusion Models in Discrete State-Spaces
• A Diffusion model for POI recommendation
• Wavelet-Packets for Deepfake Image Analysis and Detection
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
• On Fast Sampling of Diffusion Probabilistic Models
• NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates
• EPiC-GAN: Equivariant Point Cloud Generation for Particle Jets
• PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics
• A Survey on Vulnerability of Federated Learning: A Learning Algorithm Perspective
• Dynamical Regimes of Diffusion Models
• A prior regularized full waveform inversion using generative diffusion models
• A Review on Low-Dose Emission Tomography Post-Reconstruction Denoising with Neural Network Approaches
• Physics-Driven Learning for Inverse Problems in Quantum Chromodynamics
• GALA: Graph Diffusion-based Alignment with Jigsaw for Source-free Domain Adaptation
• LtU-ILI: An All-in-One Framework for Implicit Inference in Astrophysics and Cosmology
• Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions
• Fast Graph Generation via Spectral Diffusion
• On the design space between molecular mechanics and machine learning force fields
• CaloScore v2: Single-shot Calorimeter Shower Simulation with Diffusion Models
• Generative diffusion in very large dimensions
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• A Text-guided Protein Design Framework
• Mapping Dark Matter in the Milky Way using Normalizing Flows and Gaia DR3
• Controllable Data Generation by Deep Learning: A Review
• Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
• StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models
• Learning Implicit Priors for Motion Optimization
• Ensemble flow reconstruction in the atmospheric boundary layer from spatially limited measurements through latent diffusion models
• High-dimensional and Permutation Invariant Anomaly Detection
• A Domain Translation Framework with an Adversarial Denoising Diffusion Model to Generate Synthetic Datasets of Echocardiography Images
• The Landscape of Unfolding with Machine Learning
• Tailoring Machine Learning for Process Mining
• Conditional score-based diffusion models for solving inverse problems in mechanics
• Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations
• Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution
• Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution
• Bayesian Renormalization
• Score-Based Generative Models for PET Image Reconstruction
• Bi-fidelity Variational Auto-encoder for Uncertainty Quantification
• A Gentle Introduction and Tutorial on Deep Generative Models in Transportation Research
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
• 4D Facial Expression Diffusion Model
• Learned multiphysics inversion with differentiable programming and machine learning
• Non Gaussian Denoising Diffusion Models
• Deep Generative Imputation Model for Missing Not At Random Data
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• Iterative $\alpha$-(de)Blending: a Minimalist Deterministic Diffusion Model
• DiffPhase: Generative Diffusion-based STFT Phase Retrieval
• LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech
• Geometric-Facilitated Denoising Diffusion Model for 3D Molecule Generation
• Probing the Limits and Capabilities of Diffusion Models for the Anatomic Editing of Digital Twins
• Data Consistent Deep Rigid MRI Motion Correction
• Full Event Particle-Level Unfolding with Variable-Length Latent Variational Diffusion
• Score-Based Diffusion Models for Photoacoustic Tomography Image Reconstruction
• Diffusion-Based Audio Inpainting
• Diffusion-Augmented Depth Prediction with Sparse Annotations
• Integrating Dynamical Systems Modeling with Spatiotemporal scRNA-seq Data Analysis
• A Flow-based Truncated Denoising Diffusion Model for Super-resolution Magnetic Resonance Spectroscopic Imaging
• Physics-informed generative model for drug-like molecule conformers
• Modeling Partially Observed Nonlinear Dynamical Systems and Efficient Data Assimilation via Discrete-Time Conditional Gaussian Koopman Network
• PROUD: PaRetO-gUided Diffusion Model for Multi-objective Generation
• Score-matching neural networks for improved multi-band source separation
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Adaptive Monte Carlo augmented with normalizing flows
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• Back-Projection Diffusion: Solving the Wideband Inverse Scattering Problem with Diffusion Models
• Multi-Resolution Diffusion for Privacy-Sensitive Recommender Systems
• SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models
• NU-Class Net: A Novel Approach for Video Quality Enhancement
• RITA: Boost Driving Simulators with Realistic Interactive Traffic Flow
• Zero Shot Molecular Generation via Similarity Kernels
• Generative modeling of time-dependent densities via optimal transport and projection pursuit
• A Mathematical Framework for Learning Probability Distributions
• General Binding Affinity Guidance for Diffusion Models in Structure-Based Drug Design
• Detecting Modeling Bias with Continuous Time Flow Models on Weak Lensing Maps
• A Regularized Conditional GAN for Posterior Sampling in Image Recovery Problems
• Generative Priors for MRI Reconstruction Trained from Magnitude-Only Images Using Phase Augmentation
• Fokker-Planck Score Learning: Efficient Free-Energy Estimation under Periodic Boundary Conditions
• Language Guided Adversarial Purification
• Harmonic Path Integral Diffusion
• Learning Markov Random Fields for Combinatorial Structures via Sampling through Lovász Local Lemma
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
• Image Augmentation with Controlled Diffusion for Weakly-Supervised Semantic Segmentation
• Generalisation under gradient descent via deterministic PAC-Bayes
• Turbulent Injection assisted by Diffusion Models for Scale Resolving Simulations
• Bridge Diffusion Model: Bridge Chinese Text-to-Image Diffusion Model with English Communities
• U-Turn Diffusion
• Fast emulation of anisotropies induced in the cosmic microwave background by cosmic strings
• Monte Carlo guided Diffusion for Bayesian linear inverse problems
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Choose Your Diffusion: Efficient and flexible ways to accelerate the diffusion model in fast high energy physics simulation
• Distribution Preserving Source Separation With Time Frequency Predictive Models
• Efficient Kilometer-Scale Precipitation Downscaling with Conditional Wavelet Diffusion
• A Reference-Based 3D Semantic-Aware Framework for Accurate Local Facial Attribute Editing
• Nonadiabatic force matching for alchemical free-energy estimation
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs
• E2V-SDE: From Asynchronous Events to Fast and Continuous Video Reconstruction via Neural Stochastic Differential Equations
• Conditionally Strongly Log-Concave Generative Models
• Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models