Diffusion Models Beat GANs on Image Synthesis

Abstract

We show that diffusion models can achieve image sample quality superior to the current state-of-the-art generative models. We achieve this on unconditional image synthesis by finding a better architecture through a series of ablations. For conditional image synthesis, we further improve sample quality with classifier guidance: a simple, compute-efficient method for trading off diversity for fidelity using gradients from a classifier. We achieve an FID of 2.97 on ImageNet 128$\times$128, 4.59 on ImageNet 256$\times$256, and 7.72 on ImageNet 512$\times$512, and we match BigGAN-deep even with as few as 25 forward passes per sample, all while maintaining better coverage of the distribution. Finally, we find that classifier guidance combines well with upsampling diffusion models, further improving FID to 3.94 on ImageNet 256$\times$256 and 3.85 on ImageNet 512$\times$512. We release our code at https://github.com/openai/guided-diffusion

Added compute requirements, ImageNet 256$\times$256 upsampling FID and samples, DDIM guided sampler, fixed typos

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• Denoising Diffusion Probabilistic Models
• Intriguing properties of neural networks
• Learning Transferable Visual Models From Natural Language Supervision
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Improved Techniques for Training GANs
• Score-Based Generative Modeling through Stochastic Differential Equations
• Zero-Shot Text-to-Image Generation
• Generative Modeling by Estimating Gradients of the Data Distribution
• Mixed Precision Training
• Rethinking the Inception Architecture for Computer Vision
• Improved Denoising Diffusion Probabilistic Models
• cGANs with Projection Discriminator
• NVAE: A Deep Hierarchical Variational Autoencoder
• Improved Precision and Recall Metric for Assessing Generative Models
• A Note on the Inception Score
• Improved Techniques for Training Score-Based Generative Models
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Jukebox: A Generative Model for Music
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Denoising Diffusion Implicit Models
• Implicit Generation and Generalization in Energy-Based Models
• High-Fidelity Image Generation With Fewer Labels
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
• LOGAN: Latent Optimisation for Generative Adversarial Networks
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• WaveGrad: Estimating Gradients for Waveform Generation
• Adversarial score matching and improved sampling for image generation
• Variational Walkback: Learning a Transition Operator as a Stochastic Recurrent Net
• Generating Images with Sparse Representations

Cited by in corpus (158)

• Diffusion Models in Vision: A Survey
• Blended Diffusion for Text-driven Editing of Natural Images
• Cascaded Diffusion Models for High Fidelity Image Generation
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• Ten Years of Generative Adversarial Nets (GANs): A survey of the state-of-the-art
• SpaText: Spatio-Textual Representation for Controllable Image Generation
• Solving 3D Inverse Problems using Pre-trained 2D Diffusion Models
• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
• CBCT-Based Synthetic CT Image Generation Using Conditional Denoising Diffusion Probabilistic Model
• Conditional Diffusion Models for Semantic 3D Brain MRI Synthesis
• A survey on GANs for computer vision: Recent research, analysis and taxonomy
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• Crystal Diffusion Variational Autoencoder for Periodic Material Generation
• DiffEdit: Diffusion-based semantic image editing with mask guidance
• A survey of recent methods for addressing AI fairness and bias in biomedicine
• SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models
• SAR Despeckling using a Denoising Diffusion Probabilistic Model
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Vector-quantized Image Modeling with Improved VQGAN
• Denoising diffusion algorithm for inverse design of microstructures with fine-tuned nonlinear material properties
• Adaptive Latent Diffusion Model for 3D Medical Image to Image Translation: Multi-modal Magnetic Resonance Imaging Study
• Taming the Power of Diffusion Models for High-Quality Virtual Try-On with Appearance Flow
• Diffusion Probabilistic Model Based Accurate and High-Degree-of-Freedom Metasurface Inverse Design
• CLE Diffusion: Controllable Light Enhancement Diffusion Model
• A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
• LDMVFI: Video Frame Interpolation with Latent Diffusion Models
• A Diffusion model for POI recommendation
• Deep Diffusion Models for Seismic Processing
• A Systematic Survey of Regularization and Normalization in GANs
• Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
• On Fast Sampling of Diffusion Probabilistic Models
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Spatio-temporal Diffusion Point Processes
• PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics
• Learning to Efficiently Sample from Diffusion Probabilistic Models
• Maximum Likelihood Training of Score-Based Diffusion Models
• FakeNews: GAN-based generation of realistic 3D volumetric data -- A systematic review and taxonomy
• Generative Artificial Intelligence Meets Synthetic Aperture Radar: A Survey
• Denoising diffusion-based MRI to CT image translation enables automated spinal segmentation
• ControlMat: A Controlled Generative Approach to Material Capture
• SGDiff: A Style Guided Diffusion Model for Fashion Synthesis
• Guided Image Synthesis via Initial Image Editing in Diffusion Model
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Vector Quantized Diffusion Model for Text-to-Image Synthesis
• Real Risks of Fake Data: Synthetic Data, Diversity-Washing and Consent Circumvention
• A Review on Low-Dose Emission Tomography Post-Reconstruction Denoising with Neural Network Approaches
• Latent Diffusion Model for Conditional Reservoir Facies Generation
• DALLE-URBAN: Capturing the urban design expertise of large text to image transformers
• MatFuse: Controllable Material Generation with Diffusion Models
• Systematic Review on Learning-based Spectral CT
• Denoising diffusion probabilistic models for probabilistic energy forecasting
• High Perceptual Quality Wireless Image Delivery with Denoising Diffusion Models
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• FeatureNeRF: Learning Generalizable NeRFs by Distilling Foundation Models
• The Infinite Index: Information Retrieval on Generative Text-To-Image Models
• MAM-E: Mammographic synthetic image generation with diffusion models
• 360-Degree Panorama Generation from Few Unregistered NFoV Images
• EDMP: Ensemble-of-costs-guided Diffusion for Motion Planning
• Controllable Data Generation by Deep Learning: A Review
• A Domain Translation Framework with an Adversarial Denoising Diffusion Model to Generate Synthetic Datasets of Echocardiography Images
• RNDiff: Rainfall nowcasting with Condition Diffusion Model
• Ensemble flow reconstruction in the atmospheric boundary layer from spatially limited measurements through latent diffusion models
• Generative Adversarial Networks Applied to Synthetic Financial Scenarios Generation
• Conditional score-based diffusion models for solving inverse problems in mechanics
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• Guided Reconstruction with Conditioned Diffusion Models for Unsupervised Anomaly Detection in Brain MRIs
• Generative Learning of the Solution of Parametric Partial Differential Equations Using Guided Diffusion Models and Virtual Observations
• Score-Based Generative Models for PET Image Reconstruction
• Face Morphing Attack Detection with Denoising Diffusion Probabilistic Models
• 4D Facial Expression Diffusion Model
• Zero-Shot Text-Guided Object Generation with Dream Fields
• Prompting for products: Investigating design space exploration strategies for text-to-image generative models
• Multimodal Data Augmentation for Image Captioning using Diffusion Models
• Boosting GUI Prototyping with Diffusion Models
• Machine-Learning-Assisted Photonic Device Development: A Multiscale Approach from Theory to Characterization
• EASI-Tex: Edge-Aware Mesh Texturing from Single Image
• Constrained CycleGAN for Effective Generation of Ultrasound Sector Images of Improved Spatial Resolution
• LVCD: Reference-based Lineart Video Colorization with Diffusion Models
• Iterative $\alpha$-(de)Blending: a Minimalist Deterministic Diffusion Model
• ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation
• DiffMark: Diffusion-based Robust Watermark Against Deepfakes
• The stochastic digital human is now enrolling for in silico imaging trials -- Methods and tools for generating digital cohorts
• SegDiff: Image Segmentation with Diffusion Probabilistic Models
• Treatment-aware Diffusion Probabilistic Model for Longitudinal MRI Generation and Diffuse Glioma Growth Prediction
• 3D Brain and Heart Volume Generative Models: A Survey
• License Plate Super-Resolution Using Diffusion Models
• Score-based Generative Modeling in Latent Space
• LightGrad: Lightweight Diffusion Probabilistic Model for Text-to-Speech
• Data Augmentation in Earth Observation: A Diffusion Model Approach
• Balancing User Preferences by Social Networks: A Condition-Guided Social Recommendation Model for Mitigating Popularity Bias
• RetAssist: Facilitating Vocabulary Learners with Generative Images in Story Retelling Practices
• Spectrally Decomposed Diffusion Models for Generative Turbulence Recovery
• Bilateral Denoising Diffusion Models
• DRUM: Diffusion-based runoff model for probabilistic flood forecasting
• Diffiner: A Versatile Diffusion-based Generative Refiner for Speech Enhancement
• FCBoost-Net: A Generative Network for Synthesizing Multiple Collocated Outfits via Fashion Compatibility Boosting
• Perturbing Attention Gives You More Bang for the Buck: Subtle Imaging Perturbations That Efficiently Fool Customized Diffusion Models
• Frequency-Controlled Diffusion Model for Versatile Text-Guided Image-to-Image Translation
• Reflection-Equivariant Diffusion for 3D Structure Determination from Isotopologue Rotational Spectra in Natural Abundance
• Diffusion-Augmented Depth Prediction with Sparse Annotations
• A Flow-based Truncated Denoising Diffusion Model for Super-resolution Magnetic Resonance Spectroscopic Imaging
• PhenDiff: Revealing Subtle Phenotypes with Diffusion Models in Real Images
• Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models
• Diffusion-Based Audio Inpainting
• Diffusion Priors In Variational Autoencoders
• Global Search for Optimal Low Thrust Spacecraft Trajectories using Diffusion Models and the Indirect Method
• Synomaly Noise and Multi-Stage Diffusion: A Novel Approach for Unsupervised Anomaly Detection in Medical Images
• Unraveling the Potential of Diffusion Models in Small Molecule Generation
• Conditional Image Generation with Score-Based Diffusion Models
• Improved HER2 Tumor Segmentation with Subtype Balancing using Deep Generative Networks
• Inverse problem regularization with hierarchical variational autoencoders
• StereoDiffusion: Training-Free Stereo Image Generation Using Latent Diffusion Models
• CLIP4Sketch: Enhancing Sketch to Mugshot Matching through Dataset Augmentation using Diffusion Models
• Cycle Consistency-based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems
• Retinal OCT Synthesis with Denoising Diffusion Probabilistic Models for Layer Segmentation
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Denoising Diffusion Gamma Models
• ZePo: Zero-Shot Portrait Stylization with Faster Sampling
• Moser Flow: Divergence-based Generative Modeling on Manifolds
• SEEDS: Exponential SDE Solvers for Fast High-Quality Sampling from Diffusion Models
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• Generation of Probabilistic Synthetic Data for Serious Games: A Case Study on Cyberbullying
• Optimal Linear Subspace Search: Learning to Construct Fast and High-Quality Schedulers for Diffusion Models
• Comprehensive Dataset of Synthetic and Manipulated Overhead Imagery for Development and Evaluation of Forensic Tools
• ID-Booth: Identity-consistent Face Generation with Diffusion Models
• Improving Robustness using Generated Data
• Dreaming of Electrical Waves: Generative Modeling of Cardiac Excitation Waves using Diffusion Models
• DiffBlender: Composable and Versatile Multimodal Text-to-Image Diffusion Models
• Content-Aware Preserving Image Generation
• Reliable Multi-modal Medical Image-to-image Translation Independent of Pixel-wise Aligned Data
• Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation
• Learning multi-scale local conditional probability models of images
• EdiTTS: Score-based Editing for Controllable Text-to-Speech
• Content-aware Tile Generation using Exterior Boundary Inpainting
• The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization
• A Conditional Point Cloud Diffusion Model for Deformable Liver Motion Tracking Via a Single Arbitrarily-Angled X-ray Projection
• Path and Bone-Contour Regularized Unpaired MRI-to-CT Translation
• A multiscale and multicriteria Generative Adversarial Network to synthesize 1-dimensional turbulent fields
• BOTH2Hands: Inferring 3D Hands from Both Text Prompts and Body Dynamics
• Explaining the effects of non-convergent sampling in the training of Energy-Based Models
• IgCONDA-PET: Weakly-Supervised PET Anomaly Detection using Implicitly-Guided Attention-Conditional Counterfactual Diffusion Modeling -- a Multi-Center, Multi-Cancer, and Multi-Tracer Study
• Style transfer between Microscopy and Magnetic Resonance Imaging via Generative Adversarial Network in small sample size settings
• Turbulent Injection assisted by Diffusion Models for Scale Resolving Simulations
• Autoregressive Diffusion Models
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Bringing Balance to Hand Shape Classification: Mitigating Data Imbalance Through Generative Models
• Adapt and Diffuse: Sample-adaptive Reconstruction via Latent Diffusion Models
• Lane Graph Extraction from Aerial Imagery via Lane Segmentation Refinement with Diffusion Models
• Instance-Conditioned GAN
• Conditionally Strongly Log-Concave Generative Models
• Zero-Shot Translation using Diffusion Models
• Improving Compositionality of Neural Networks by Decoding Representations to Inputs
• Ultra-Fast Generation of Air Shower Images for Imaging Air Cherenkov Telescopes using Generative Adversarial Networks
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Neural Differential Appearance Equations
• Pre to Post-Treatment Glioblastoma MRI Prediction using a Latent Diffusion Model