D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
arXiv:2106.06819
Abstract
Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study.
References in corpus (55)
- Conditional Generative Adversarial Nets
- A Simple Framework for Contrastive Learning of Visual Representations
- Denoising Diffusion Probabilistic Models
- Learning Transferable Visual Models From Natural Language Supervision
- Deep Residual Learning for Image Recognition
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Bootstrap your own latent: A new approach to self-supervised Learning
- Spectral Normalization for Generative Adversarial Networks
- Language Models are Few-Shot Learners
- Neural Discrete Representation Learning
- Generative Adversarial Text to Image Synthesis
- Diffusion Models Beat GANs on Image Synthesis
- Improved Baselines with Momentum Contrastive Learning
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Deep Unsupervised Learning using Nonequilibrium Thermodynamics
- NICE: Non-linear Independent Components Estimation
- Score-Based Generative Modeling through Stochastic Differential Equations
- Zero-Shot Text-to-Image Generation
- Generative Modeling by Estimating Gradients of the Data Distribution
- Momentum Contrast for Unsupervised Visual Representation Learning
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Density estimation using Real NVP
- Adversarial Feature Learning
- Adversarially Learned Inference
- Neural Ordinary Differential Equations
- Exploring Simple Siamese Representation Learning
- NVAE: A Deep Hierarchical Variational Autoencoder
- InfoVAE: Information Maximizing Variational Autoencoders
- Neural Photo Editing with Introspective Adversarial Networks
- Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation
- Wasserstein Auto-Encoders
- Learning in Implicit Generative Models
- Swapping Autoencoder for Deep Image Manipulation
- Towards Deeper Understanding of Variational Autoencoding Models
- Jukebox: A Generative Model for Music
- Generating Diverse High-Fidelity Images with VQ-VAE-2
- Denoising Diffusion Implicit Models
- Implicit Generation and Generalization in Energy-Based Models
- Diagnosing and Enhancing VAE Models
- Improving Variational Auto-Encoders using Householder Flow
- Generating Images from Captions with Attention
- FIGR: Few-shot Image Generation with Reptile
- Learning Controllable Fair Representations
- VAE with a VampPrior
- Distribution Matching in Variational Inference
- Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
- Invertible generative models for inverse problems: mitigating representation error and dataset bias
- Fixing a Broken ELBO
- Generative Adversarial Transformers
- Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
- Multi-label Contrastive Predictive Coding
- Symbolic Music Generation with Diffusion Models
- A Contrastive Learning Approach for Training Variational Autoencoder Priors
- Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation