Maximum Likelihood Training of Score-Based Diffusion Models

Abstract

Score-based diffusion models synthesize samples by reversing a stochastic process that diffuses data to noise, and are trained by minimizing a weighted combination of score matching losses. The log-likelihood of score-based diffusion models can be tractably computed through a connection to continuous normalizing flows, but log-likelihood is not directly optimized by the weighted combination of score matching losses. We show that for a specific weighting scheme, the objective upper bounds the negative log-likelihood, thus enabling approximate maximum likelihood training of score-based diffusion models. We empirically observe that maximum likelihood training consistently improves the likelihood of score-based diffusion models across multiple datasets, stochastic processes, and model architectures. Our best models achieve negative log-likelihoods of 2.83 and 3.76 bits/dim on CIFAR-10 and ImageNet 32x32 without any data augmentation, on a par with state-of-the-art autoregressive models on these tasks.

arXiv commentNeurIPS 2021 (Spotlight)

References in corpus (22)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Diffusion Models Beat GANs on Image Synthesis
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• NICE: Non-linear Independent Components Estimation
• Generative Modeling by Estimating Gradients of the Data Distribution
• Generating Long Sequences with Sparse Transformers
• Improved Denoising Diffusion Probabilistic Models
• Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design
• Scalable Gradients for Stochastic Differential Equations
• DiffWave: A Versatile Diffusion Model for Audio Synthesis
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• A Study of Face Obfuscation in ImageNet
• Interpretation and Generalization of Score Matching
• Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimation
• Neural Stochastic Differential Equations: Deep Latent Gaussian Models in the Diffusion Limit
• Practical Lossless Compression with Latent Variables using Bits Back Coding
• Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech
• Compression with Flows via Local Bits-Back Coding
• VFlow: More Expressive Generative Flows with Variational Data Augmentation
• Consistency Regularization for Variational Auto-Encoders

Cited by in corpus (12)

• Variational Diffusion Models
• SGDiff: A Style Guided Diffusion Model for Fashion Synthesis
• Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask
• Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
• Score-Based Generative Models for PET Image Reconstruction
• How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
• Score-based Generative Modeling in Latent Space
• Conditional Image Generation with Score-Based Diffusion Models
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• Likelihood Training of Schrödinger Bridge using Forward-Backward SDEs Theory
• Density Ratio Estimation via Infinitesimal Classification
• Score-Based Generative Classifiers