Learning Compositional Visual Concepts with Mutual Consistency

Abstract

Compositionality of semantic concepts in image synthesis and analysis is appealing as it can help in decomposing known and generatively recomposing unknown data. For instance, we may learn concepts of changing illumination, geometry or albedo of a scene, and try to recombine them to generate physically meaningful, but unseen data for training and testing. In practice however we often do not have samples from the joint concept space available: We may have data on illumination change in one data set and on geometric change in another one without complete overlap. We pose the following question: How can we learn two or more concepts jointly from different data sets with mutual consistency where we do not have samples from the full joint space? We present a novel answer in this paper based on cyclic consistency over multiple concepts, represented individually by generative adversarial networks (GANs). Our method, ConceptGAN, can be understood as a drop in for data augmentation to improve resilience for real world applications. Qualitative and quantitative evaluations demonstrate its efficacy in generating semantically meaningful images, as well as one shot face verification as an example application.

10 pages, 8 figures, 4 tables, CVPR 2018

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conditional Generative Adversarial Nets
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Deep Convolutional Inverse Graphics Network
• Invertible Conditional GANs for image editing
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Age Progression/Regression by Conditional Adversarial Autoencoder
• Tell Me Where to Look: Guided Attention Inference Network
• Unsupervised Visual Attribute Transfer with Reconfigurable Generative Adversarial Networks
• Generative Semantic Manipulation with Contrasting GAN
• Attribute-Guided Face Generation Using Conditional CycleGAN

Cited by in corpus (2)

• Tell Me Where to Look: Guided Attention Inference Network
• Translating and Segmenting Multimodal Medical Volumes with Cycle- and Shape-Consistency Generative Adversarial Network