Instance-Conditioned GAN
arXiv:2109.05070
Abstract
Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions of datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. In this paper, we take inspiration from kernel density estimation techniques and introduce a non-parametric approach to modeling distributions of complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors, and introduce a model, called instance-conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results on ImageNet and COCO-Stuff show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN can effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, we extend IC-GAN to the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet; while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available at https://github.com/facebookresearch/ic_gan.
Accepted at NeurIPS2021
References in corpus (8)
- Conditional Generative Adversarial Nets
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Diffusion Models Beat GANs on Image Synthesis
- The Deepfake Detection Challenge (DFDC) Preview Dataset
- High-Fidelity Image Generation With Fewer Labels
- Large-Scale Long-Tailed Recognition in an Open World
- Generating unseen complex scenes: are we there yet?
- Self-labeled Conditional GANs