Controlling generative models with continuous factors of variations

Abstract

Recent deep generative models are able to provide photo-realistic images as well as visual or textual content embeddings useful to address various tasks of computer vision and natural language processing. Their usefulness is nevertheless often limited by the lack of control over the generative process or the poor understanding of the learned representation. To overcome these major issues, very recent work has shown the interest of studying the semantics of the latent space of generative models. In this paper, we propose to advance on the interpretability of the latent space of generative models by introducing a new method to find meaningful directions in the latent space of any generative model along which we can move to control precisely specific properties of the generated image like the position or scale of the object in the image. Our method does not require human annotations and is particularly well suited for the search of directions encoding simple transformations of the generated image, such as translation, zoom or color variations. We demonstrate the effectiveness of our method qualitatively and quantitatively, both for GANs and variational auto-encoders.

Accepted as a poster presentation at the International Conference for Learning Representations (ICLR), 2020

References in corpus (16)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Deeply supervised salient object detection with short connections
• Autoencoding beyond pixels using a learned similarity metric
• Deep multi-scale video prediction beyond mean square error
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• Modulating early visual processing by language
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Generative Visual Manipulation on the Natural Image Manifold
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• On the "steerability" of generative adversarial networks
• Inverting The Generator Of A Generative Adversarial Network (II)

Cited by in corpus (23)

• GANSpace: Discovering Interpretable GAN Controls
• Unsupervised Discovery of Interpretable Directions in the GAN Latent Space
• Fixing Data Augmentation to Improve Adversarial Robustness
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis
• Closed-Form Factorization of Latent Semantics in GANs
• A Survey on Neural Network Interpretability
• Counterfactual Generative Networks
• GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
• High-Fidelity GAN Inversion for Image Attribute Editing
• GANravel: User-Driven Direction Disentanglement in Generative Adversarial Networks
• GAN "Steerability" without optimization
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Do Not Escape From the Manifold: Discovering the Local Coordinates on the Latent Space of GANs
• Improving Robustness using Generated Data
• Towards Principled Disentanglement for Domain Generalization
• Ensembling with Deep Generative Views
• Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis
• Disentangled Face Attribute Editing via Instance-Aware Latent Space Search
• Low-Rank Subspaces in GANs
• Surrogate Gradient Field for Latent Space Manipulation
• SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editing