D2C: Diffusion-Denoising Models for Few-shot Conditional Generation

Abstract

Conditional generative models of high-dimensional images have many applications, but supervision signals from conditions to images can be expensive to acquire. This paper describes Diffusion-Decoding models with Contrastive representations (D2C), a paradigm for training unconditional variational autoencoders (VAEs) for few-shot conditional image generation. D2C uses a learned diffusion-based prior over the latent representations to improve generation and contrastive self-supervised learning to improve representation quality. D2C can adapt to novel generation tasks conditioned on labels or manipulation constraints, by learning from as few as 100 labeled examples. On conditional generation from new labels, D2C achieves superior performance over state-of-the-art VAEs and diffusion models. On conditional image manipulation, D2C generations are two orders of magnitude faster to produce over StyleGAN2 ones and are preferred by 50% - 60% of the human evaluators in a double-blind study.

References in corpus (19)

• Conditional Generative Adversarial Nets
• Learning Transferable Visual Models From Natural Language Supervision
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Bootstrap your own latent: A new approach to self-supervised Learning
• Language Models are Few-Shot Learners
• Diffusion Models Beat GANs on Image Synthesis
• NICE: Non-linear Independent Components Estimation
• Score-Based Generative Modeling through Stochastic Differential Equations
• Zero-Shot Text-to-Image Generation
• Exploring Simple Siamese Representation Learning
• Neural Photo Editing with Introspective Adversarial Networks
• Learning in Implicit Generative Models
• Towards Deeper Understanding of Variational Autoencoding Models
• Jukebox: A Generative Model for Music
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Improving Variational Auto-Encoders using Householder Flow
• FIGR: Few-shot Image Generation with Reptile
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation