Tackling the Generative Learning Trilemma with Denoising Diffusion GANs

Abstract

A wide variety of deep generative models has been developed in the past decade. Yet, these models often struggle with simultaneously addressing three key requirements including: high sample quality, mode coverage, and fast sampling. We call the challenge imposed by these requirements the generative learning trilemma, as the existing models often trade some of them for others. Particularly, denoising diffusion models have shown impressive sample quality and diversity, but their expensive sampling does not yet allow them to be applied in many real-world applications. In this paper, we argue that slow sampling in these models is fundamentally attributed to the Gaussian assumption in the denoising step which is justified only for small step sizes. To enable denoising with large steps, and hence, to reduce the total number of denoising steps, we propose to model the denoising distribution using a complex multimodal distribution. We introduce denoising diffusion generative adversarial networks (denoising diffusion GANs) that model each denoising step using a multimodal conditional GAN. Through extensive evaluations, we show that denoising diffusion GANs obtain sample quality and diversity competitive with original diffusion models while being 2000$\times$ faster on the CIFAR-10 dataset. Compared to traditional GANs, our model exhibits better mode coverage and sample diversity. To the best of our knowledge, denoising diffusion GAN is the first model that reduces sampling cost in diffusion models to an extent that allows them to be applied to real-world applications inexpensively. Project page and code can be found at https://nvlabs.github.io/denoising-diffusion-gan

arXiv commentICLR 2022 (Spotlight)

Cited by in corpus (18)

• Diffusion Models in Vision: A Survey
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Adaptive Diffusion Priors for Accelerated MRI Reconstruction
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• Diffusion Probabilistic Model Based Accurate and High-Degree-of-Freedom Metasurface Inverse Design
• Copyright Protection and Accountability of Generative AI:Attack, Watermarking and Attribution
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• A Domain Translation Framework with an Adversarial Denoising Diffusion Model to Generate Synthetic Datasets of Echocardiography Images
• Score-Based Generative Models for PET Image Reconstruction
• A Gentle Introduction and Tutorial on Deep Generative Models in Transportation Research
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
• Waving Goodbye to Low-Res: A Diffusion-Wavelet Approach for Image Super-Resolution
• Spectrally Decomposed Diffusion Models for Generative Turbulence Recovery
• SeeDS: Semantic Separable Diffusion Synthesizer for Zero-shot Food Detection
• Discriminative Hamiltonian Variational Autoencoder for Accurate Tumor Segmentation in Data-Scarce Regimes
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• Turbulent Injection assisted by Diffusion Models for Scale Resolving Simulations