GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data

Abstract

Object Transfiguration replaces an object in an image with another object from a second image. For example it can perform tasks like "putting exactly those eyeglasses from image A on the nose of the person in image B". Usage of exemplar images allows more precise specification of desired modifications and improves the diversity of conditional image generation. However, previous methods that rely on feature space operations, require paired data and/or appearance models for training or disentangling objects from background. In this work, we propose a model that can learn object transfiguration from two unpaired sets of images: one set containing images that "have" that kind of object, and the other set being the opposite, with the mild constraint that the objects be located approximately at the same place. For example, the training data can be one set of reference face images that have eyeglasses, and another set of images that have not, both of which spatially aligned by face landmarks. Despite the weak 0/1 labels, our model can learn an "eyeglasses" subspace that contain multiple representatives of different types of glasses. Consequently, we can perform fine-grained control of generated images, like swapping the glasses in two images by swapping the projected components in the "eyeglasses" subspace, to create novel images of people wearing eyeglasses. Overall, our deterministic generative model learns disentangled attribute subspaces from weakly labeled data by adversarial training. Experiments on CelebA and Multi-PIE datasets validate the effectiveness of the proposed model on real world data, in generating images with specified eyeglasses, smiling, hair styles, and lighting conditions etc. The code is available online.

Github: https://github.com/Prinsphield/GeneGAN

References in corpus (11)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Striving for Simplicity: The All Convolutional Net
• Deep Convolutional Inverse Graphics Network
• Invertible Conditional GANs for image editing
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Better Mixing via Deep Representations
• The Variational Fair Autoencoder
• Discovering Hidden Factors of Variation in Deep Networks
• Generative Visual Manipulation on the Natural Image Manifold
• Deep Manifold Traversal: Changing Labels with Convolutional Features
• Learning to Generate Chairs, Tables and Cars with Convolutional Networks

Cited by in corpus (12)

• How Generative Adversarial Networks and Their Variants Work: An Overview
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Deep Identity-aware Transfer of Facial Attributes
• Learning Disentangled Representations in the Imaging Domain
• Recent Progress of Face Image Synthesis
• ChildPredictor: A Child Face Prediction Framework with Disentangled Learning
• MulGAN: Facial Attribute Editing by Exemplar
• Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation
• Instance-level Facial Attributes Transfer with Geometry-Aware Flow
• Covering up bias in CelebA-like datasets with Markov blankets: A post-hoc cure for attribute prior avoidance
• Exemplar-based Generative Facial Editing
• ClsGAN: Selective Attribute Editing Model Based On Classification Adversarial Network