Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed

Abstract

Iterative generative models, such as noise conditional score networks and denoising diffusion probabilistic models, produce high quality samples by gradually denoising an initial noise vector. However, their denoising process has many steps, making them 2-3 orders of magnitude slower than other generative models such as GANs and VAEs. In this paper, we establish a novel connection between knowledge distillation and image generation with a technique that distills a multi-step denoising process into a single step, resulting in a sampling speed similar to other single-step generative models. Our Denoising Student generates high quality samples comparable to GANs on the CIFAR-10 and CelebA datasets, without adversarial training. We demonstrate that our method scales to higher resolutions through experiments on 256 x 256 LSUN. Code and checkpoints are available at https://github.com/tcl9876/Denoising_Student

arXiv comment20 pages, 13 figures

References in corpus (16)

• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Denoising Diffusion Probabilistic Models
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Improved Techniques for Training GANs
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Score-Based Generative Modeling through Stochastic Differential Equations
• Training Generative Adversarial Networks with Limited Data
• NVAE: A Deep Hierarchical Variational Autoencoder
• Denoising Diffusion Implicit Models
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models
• Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognition

Cited by in corpus (9)

• Diffusion Models Beat GANs on Image Synthesis
• DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models
• Diffusion Models, Image Super-Resolution And Everything: A Survey
• Diffusion Schrödinger Bridge with Applications to Score-Based Generative Modeling
• Noise Estimation for Generative Diffusion Models
• Latent Denoising Diffusion GAN: Faster sampling, Higher image quality
• Score-based Generative Modeling in Latent Space
• Bilateral Denoising Diffusion Models
• Fast Inference in Denoising Diffusion Models via MMD Finetuning