InstaGAN: Instance-aware Image-to-Image Translation

Abstract

Unsupervised image-to-image translation has gained considerable attention due to the recent impressive progress based on generative adversarial networks (GANs). However, previous methods often fail in challenging cases, in particular, when an image has multiple target instances and a translation task involves significant changes in shape, e.g., translating pants to skirts in fashion images. To tackle the issues, we propose a novel method, coined instance-aware GAN (InstaGAN), that incorporates the instance information (e.g., object segmentation masks) and improves multi-instance transfiguration. The proposed method translates both an image and the corresponding set of instance attributes while maintaining the permutation invariance property of the instances. To this end, we introduce a context preserving loss that encourages the network to learn the identity function outside of target instances. We also propose a sequential mini-batch inference/training technique that handles multiple instances with a limited GPU memory and enhances the network to generalize better for multiple instances. Our comparative evaluation demonstrates the effectiveness of the proposed method on different image datasets, in particular, in the aforementioned challenging cases. Code and results are available in https://github.com/sangwoomo/instagan

Accepted to ICLR 2019. High resolution images are available in https://github.com/sangwoomo/instagan

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Self-Attention Generative Adversarial Networks
• Unsupervised Cross-Domain Image Generation
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Unsupervised Attention-guided Image to Image Translation

Cited by in corpus (20)

• Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery
• Identity-Aware CycleGAN for Face Photo-Sketch Synthesis and Recognition
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Image-to-Image Translation: Methods and Applications
• M2m: Imbalanced Classification via Major-to-minor Translation
• Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Maps
• Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation
• Cross-Domain Adaptation for Animal Pose Estimation
• GANHopper: Multi-Hop GAN for Unsupervised Image-to-Image Translation
• PT2PC: Learning to Generate 3D Point Cloud Shapes from Part Tree Conditions
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• ManiGAN: Text-Guided Image Manipulation
• Global and Local Alignment Networks for Unpaired Image-to-Image Translation
• Domain-Specific Mappings for Generative Adversarial Style Transfer
• Attribute-Guided Sketch Generation
• Integrating Categorical Semantics into Unsupervised Domain Translation
• Semantic Editing On Segmentation Map Via Multi-Expansion Loss
• Memory-guided Unsupervised Image-to-image Translation
• Neural Wireframe Renderer: Learning Wireframe to Image Translations
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey