Classifier-Free Diffusion Guidance

Abstract

Classifier guidance is a recently introduced method to trade off mode coverage and sample fidelity in conditional diffusion models post training, in the same spirit as low temperature sampling or truncation in other types of generative models. Classifier guidance combines the score estimate of a diffusion model with the gradient of an image classifier and thereby requires training an image classifier separate from the diffusion model. It also raises the question of whether guidance can be performed without a classifier. We show that guidance can be indeed performed by a pure generative model without such a classifier: in what we call classifier-free guidance, we jointly train a conditional and an unconditional diffusion model, and we combine the resulting conditional and unconditional score estimates to attain a trade-off between sample quality and diversity similar to that obtained using classifier guidance.

arXiv commentA short version of this paper appeared in the NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications: https://openreview.net/pdf?id=qw8AKxfYbI

Cited by in corpus (45)

• Diffusion Models in Vision: A Survey
• SpaText: Spatio-Textual Representation for Controllable Image Generation
• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• DiffEdit: Diffusion-based semantic image editing with mask guidance
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Diffusion Probabilistic Model Based Accurate and High-Degree-of-Freedom Metasurface Inverse Design
• NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates
• PC-JeDi: Diffusion for Particle Cloud Generation in High Energy Physics
• Generative Artificial Intelligence Meets Synthetic Aperture Radar: A Survey
• Denoising diffusion-based MRI to CT image translation enables automated spinal segmentation
• SGDiff: A Style Guided Diffusion Model for Fashion Synthesis
• MatFuse: Controllable Material Generation with Diffusion Models
• DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation
• 360-Degree Panorama Generation from Few Unregistered NFoV Images
• Controllable Data Generation by Deep Learning: A Review
• Unstoppable Attack: Label-Only Model Inversion via Conditional Diffusion Model
• Fast Diffusion EM: a diffusion model for blind inverse problems with application to deconvolution
• Score-Based Generative Models for PET Image Reconstruction
• UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons
• Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data
• Counterfactual Explanations for Medical Image Classification and Regression using Diffusion Autoencoder
• ElasticDiffusion: Training-free Arbitrary Size Image Generation through Global-Local Content Separation
• Probing the Limits and Capabilities of Diffusion Models for the Anatomic Editing of Digital Twins
• Balancing User Preferences by Social Networks: A Condition-Guided Social Recommendation Model for Mitigating Popularity Bias
• DRUM: Diffusion-based runoff model for probabilistic flood forecasting
• Spectrally Decomposed Diffusion Models for Generative Turbulence Recovery
• Reflection-Equivariant Diffusion for 3D Structure Determination from Isotopologue Rotational Spectra in Natural Abundance
• Diffusion-Augmented Depth Prediction with Sparse Annotations
• Unraveling the Potential of Diffusion Models in Small Molecule Generation
• Global Search for Optimal Low Thrust Spacecraft Trajectories using Diffusion Models and the Indirect Method
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• Algebraic Language Models for Inverse Design of Metamaterials via Diffusion Transformers
• A Bayesian Flow Network Framework for Chemistry Tasks
• Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation
• Conditional Distribution Modelling for Few-Shot Image Synthesis with Diffusion Models
• Content-Aware Preserving Image Generation
• Turbulent Injection assisted by Diffusion Models for Scale Resolving Simulations
• Model See Model Do: Speech-Driven Facial Animation with Style Control
• IgCONDA-PET: Weakly-Supervised PET Anomaly Detection using Implicitly-Guided Attention-Conditional Counterfactual Diffusion Modeling -- a Multi-Center, Multi-Cancer, and Multi-Tracer Study
• ESCT3D: Efficient and Selectively Controllable Text-Driven 3D Content Generation with Gaussian Splatting
• Conditional Diffusion Guided Knowledge Transfer for Multi-Domain Knowledge Graph Completion
• A Diffusion-based Data Generator for Training Object Recognition Models in Ultra-Range Distance
• DiTango: Cost-Effective Parallel Diffusion Generation with Selective Attention State Reuse
• Fidelity-preserving enhancement of ptychography with foundational text-to-image models