Ensembling with Deep Generative Views
arXiv:2104.14551
Abstract
Recent generative models can synthesize "views" of artificial images that mimic real-world variations, such as changes in color or pose, simply by learning from unlabeled image collections. Here, we investigate whether such views can be applied to real images to benefit downstream analysis tasks such as image classification. Using a pretrained generator, we first find the latent code corresponding to a given real input image. Applying perturbations to the code creates natural variations of the image, which can then be ensembled together at test-time. We use StyleGAN2 as the source of generative augmentations and investigate this setup on classification tasks involving facial attributes, cat faces, and cars. Critically, we find that several design decisions are required towards making this process work; the perturbation procedure, weighting between the augmentations and original image, and training the classifier on synthesized images can all impact the result. Currently, we find that while test-time ensembling with GAN-based augmentations can offer some small improvements, the remaining bottlenecks are the efficiency and accuracy of the GAN reconstructions, coupled with classifier sensitivities to artifacts in GAN-generated images.
arXiv commentCVPR 2021 camera ready version; code available at https://github.com/chail/gan-ensembling
References in corpus (20)
- Adam: A Method for Stochastic Optimization
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Self-Attention Generative Adversarial Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Exploring the Landscape of Spatial Robustness
- Neural Photo Editing with Introspective Adversarial Networks
- Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
- GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
- Classification Accuracy Score for Conditional Generative Models
- The Robust Manifold Defense: Adversarial Training using Generative Models
- Controlling generative models with continuous factors of variations
- APE-GAN: Adversarial Perturbation Elimination with GAN
- Closed-Form Factorization of Latent Semantics in GANs
- Unsupervised Histopathology Image Synthesis
- Using latent space regression to analyze and leverage compositionality in GANs
- Counterfactual Generative Networks
- Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Space
- StyleGAN2 Distillation for Feed-forward Image Manipulation
- Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering