Instance-Conditioned GAN

Abstract

Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions of datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. In this paper, we take inspiration from kernel density estimation techniques and introduce a non-parametric approach to modeling distributions of complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors, and introduce a model, called instance-conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results on ImageNet and COCO-Stuff show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN can effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, we extend IC-GAN to the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet; while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available at https://github.com/facebookresearch/ic_gan.

arXiv commentAccepted at NeurIPS2021

References in corpus (15)

• Conditional Generative Adversarial Nets
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Self-Attention Generative Adversarial Networks
• Diffusion Models Beat GANs on Image Synthesis
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Unrolled Generative Adversarial Networks
• Improved Precision and Recall Metric for Assessing Generative Models
• The Deepfake Detection Challenge (DFDC) Preview Dataset
• High-Fidelity Image Generation With Fewer Labels
• Large-Scale Long-Tailed Recognition in an Open World
• Class-Splitting Generative Adversarial Networks
• Mixed batches and symmetric discriminators for GAN training
• Generating unseen complex scenes: are we there yet?
• Self-labeled Conditional GANs