Variational Diffusion Models

Abstract

Diffusion-based generative models have demonstrated a capacity for perceptually impressive synthesis, but can they also be great likelihood-based models? We answer this in the affirmative, and introduce a family of diffusion-based generative models that obtain state-of-the-art likelihoods on standard image density estimation benchmarks. Unlike other diffusion-based models, our method allows for efficient optimization of the noise schedule jointly with the rest of the model. We show that the variational lower bound (VLB) simplifies to a remarkably short expression in terms of the signal-to-noise ratio of the diffused data, thereby improving our theoretical understanding of this model class. Using this insight, we prove an equivalence between several models proposed in the literature. In addition, we show that the continuous-time VLB is invariant to the noise schedule, except for the signal-to-noise ratio at its endpoints. This enables us to learn a noise schedule that minimizes the variance of the resulting VLB estimator, leading to faster optimization. Combining these advances with architectural improvements, we obtain state-of-the-art likelihoods on image density estimation benchmarks, outperforming autoregressive models that have dominated these benchmarks for many years, with often significantly faster optimization. In addition, we show how to use the model as part of a bits-back compression scheme, and demonstrate lossless compression rates close to the theoretical optimum. Code is available at https://github.com/google-research/vdm .

arXiv commentPublished at NeurIPS'21

References in corpus (22)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Denoising Diffusion Probabilistic Models
• Pixel Recurrent Neural Networks
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• Generative Modeling by Estimating Gradients of the Data Distribution
• Generating Long Sequences with Sparse Transformers
• Improved Denoising Diffusion Probabilistic Models
• NVAE: A Deep Hierarchical Variational Autoencoder
• Image Transformer
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Denoising Diffusion Implicit Models
• Asymmetric numeral systems
• Bit-Swap: Recursive Bits-Back Coding for Lossless Compression with Hierarchical Latent Variables
• Maximum Likelihood Training of Score-Based Diffusion Models
• Practical Lossless Compression with Latent Variables using Bits Back Coding
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Compression with Flows via Local Bits-Back Coding
• Score-based Generative Modeling in Latent Space
• Consistency Regularization for Variational Auto-Encoders

Cited by in corpus (32)

• Diffusion Models in Vision: A Survey
• StoRM: A Diffusion-based Stochastic Regeneration Model for Speech Enhancement and Dereverberation
• Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models
• NU-Wave 2: A General Neural Audio Upsampling Model for Various Sampling Rates
• Artificial Intelligence and Deep Learning Algorithms for Epigenetic Sequence Analysis: A Review for Epigeneticists and AI Experts
• A Variational Perspective on Diffusion-Based Generative Models and Score Matching
• Fast Graph Generation via Spectral Diffusion
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
• Conditioning and Sampling in Variational Diffusion Models for Speech Super-Resolution
• Bayesian Renormalization
• PanoGAN A Deep Generative Model for Panoramic Dental Radiographs
• SegDiff: Image Segmentation with Diffusion Probabilistic Models
• Score-based Generative Modeling in Latent Space
• Geometric-Facilitated Denoising Diffusion Model for 3D Molecule Generation
• Densely connected normalizing flows
• Score-Based Diffusion Models for Photoacoustic Tomography Image Reconstruction
• Probabilistic Precision and Recall Towards Reliable Evaluation of Generative Models
• OverFlow: Putting flows on top of neural transducers for better TTS
• Probabilistic Programming with Programmable Variational Inference
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• Denoising Diffusion Gamma Models
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects
• Density Ratio Estimation via Infinitesimal Classification
• Score-Based Generative Classifiers
• Resampling Base Distributions of Normalizing Flows
• Autoregressive Diffusion Models
• Partition and Code: learning how to compress graphs