On the "steerability" of generative adversarial networks

Abstract

An open secret in contemporary machine learning is that many models work beautifully on standard benchmarks but fail to generalize outside the lab. This has been attributed to biased training data, which provide poor coverage over real world events. Generative models are no exception, but recent advances in generative adversarial networks (GANs) suggest otherwise - these models can now synthesize strikingly realistic and diverse images. Is generative modeling of photos a solved problem? We show that although current GANs can fit standard datasets very well, they still fall short of being comprehensive models of the visual manifold. In particular, we study their ability to fit simple transformations such as camera movements and color changes. We find that the models reflect the biases of the datasets on which they are trained (e.g., centered objects), but that they also exhibit some capacity for generalization: by "steering" in latent space, we can shift the distribution while still creating realistic images. We hypothesize that the degree of distributional shift is related to the breadth of the training data distribution. Thus, we conduct experiments to quantify the limits of GAN transformations and introduce techniques to mitigate the problem. Code is released on our project page: https://ali-design.github.io/gan_steerability/

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Gauge Equivariant Convolutional Networks and the Icosahedral CNN
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Interpreting the Latent Space of GANs for Semantic Face Editing
• Image Counterfactual Sensitivity Analysis for Detecting Unintended Bias
• Flat Metric Minimization with Applications in Generative Modeling

Cited by in corpus (9)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Controlling generative models with continuous factors of variations
• CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
• StyleGAN2 Distillation for Feed-forward Image Manipulation
• Attribute-specific Control Units in StyleGAN for Fine-grained Image Manipulation
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• In-Domain GAN Inversion for Real Image Editing
• Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation
• Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation