Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency

Abstract

Image-to-image translation has recently received significant attention due to advances in deep learning. Most works focus on learning either a one-to-one mapping in an unsupervised way or a many-to-many mapping in a supervised way. However, a more practical setting is many-to-many mapping in an unsupervised way, which is harder due to the lack of supervision and the complex inner- and cross-domain variations. To alleviate these issues, we propose the Exemplar Guided & Semantically Consistent Image-to-image Translation (EGSC-IT) network which conditions the translation process on an exemplar image in the target domain. We assume that an image comprises of a content component which is shared across domains, and a style component specific to each domain. Under the guidance of an exemplar from the target domain we apply Adaptive Instance Normalization to the shared content component, which allows us to transfer the style information of the target domain to the source domain. To avoid semantic inconsistencies during translation that naturally appear due to the large inner- and cross-domain variations, we introduce the concept of feature masks that provide coarse semantic guidance without requiring the use of any semantic labels. Experimental results on various datasets show that EGSC-IT does not only translate the source image to diverse instances in the target domain, but also preserves the semantic consistency during the process.

arXiv commentTo appear in ICLR 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Toward Multimodal Image-to-Image Translation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Deep Photo Style Transfer
• A Closed-form Solution to Photorealistic Image Stylization
• Conditional Image-to-Image Translation
• DA-GAN: Instance-level Image Translation by Deep Attention Generative Adversarial Networks (with Supplementary Materials)

Cited by in corpus (12)

• Progressive Pose Attention Transfer for Person Image Generation
• Example-Guided Style Consistent Image Synthesis from Semantic Labeling
• Progressive and Aligned Pose Attention Transfer for Person Image Generation
• A Novel BiLevel Paradigm for Image-to-Image Translation
• Mask Based Unsupervised Content Transfer
• Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision
• EFANet: Exchangeable Feature Alignment Network for Arbitrary Style Transfer
• Example-Guided Scene Image Synthesis using Masked Spatial-Channel Attention and Patch-Based Self-Supervision
• Learning Disentangled Representations of Satellite Image Time Series
• Online Exemplar Fine-Tuning for Image-to-Image Translation
• DLGAN: Disentangling Label-Specific Fine-Grained Features for Image Manipulation
• Separating Content and Style for Unsupervised Image-to-Image Translation