Multimodal Unsupervised Image-to-Image Translation

Abstract

Unsupervised image-to-image translation is an important and challenging problem in computer vision. Given an image in the source domain, the goal is to learn the conditional distribution of corresponding images in the target domain, without seeing any pairs of corresponding images. While this conditional distribution is inherently multimodal, existing approaches make an overly simplified assumption, modeling it as a deterministic one-to-one mapping. As a result, they fail to generate diverse outputs from a given source domain image. To address this limitation, we propose a Multimodal Unsupervised Image-to-image Translation (MUNIT) framework. We assume that the image representation can be decomposed into a content code that is domain-invariant, and a style code that captures domain-specific properties. To translate an image to another domain, we recombine its content code with a random style code sampled from the style space of the target domain. We analyze the proposed framework and establish several theoretical results. Extensive experiments with comparisons to the state-of-the-art approaches further demonstrates the advantage of the proposed framework. Moreover, our framework allows users to control the style of translation outputs by providing an example style image. Code and pretrained models are available at https://github.com/nvlabs/MUNIT

arXiv commentAccepted by ECCV 2018

References in corpus (27)

• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Toward Multimodal Image-to-Image Translation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Energy-based Generative Adversarial Network
• A Learned Representation For Artistic Style
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Adversarial Feature Learning
• Adversarially Learned Inference
• Style Transfer from Non-Parallel Text by Cross-Alignment
• Domain Separation Networks
• Unsupervised Image-to-Image Translation Networks
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Unsupervised Cross-Domain Image Generation
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Revisiting Batch Normalization For Practical Domain Adaptation
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Holistically-Nested Edge Detection
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Fast Patch-based Style Transfer of Arbitrary Style
• Unsupervised Learning of Disentangled Representations from Video
• LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
• One-Sided Unsupervised Domain Mapping
• A Closed-form Solution to Photorealistic Image Stylization
• ZM-Net: Real-time Zero-shot Image Manipulation Network
• Unsupervised Multi-Domain Image Translation with Domain-Specific Encoders/Decoders

Cited by in corpus (75)

• U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
• Embracing Imperfect Datasets: A Review of Deep Learning Solutions for Medical Image Segmentation
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Disentangled Representation Learning in Cardiac Image Analysis
• Video-to-Video Synthesis
• Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
• Neural Style Transfer: A Review
• Image-to-image translation for cross-domain disentanglement
• UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
• Domain Stylization: A Strong, Simple Baseline for Synthetic to Real Image Domain Adaptation
• Bridging the Gap between Events and Frames through Unsupervised Domain Adaptation
• Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving
• Generative adversarial interpolative autoencoding: adversarial training on latent space interpolations encourage convex latent distributions
• Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks
• Text as Neural Operator: Image Manipulation by Text Instruction
• Cali-Sketch: Stroke Calibration and Completion for High-Quality Face Image Generation from Human-Like Sketches
• Perceptual Adversarial Robustness: Defense Against Unseen Threat Models
• Unsupervised Two-Stage Anomaly Detection
• CariGAN: Caricature Generation through Weakly Paired Adversarial Learning
• Dynamic Instance Normalization for Arbitrary Style Transfer
• Guided Image-to-Image Translation with Bi-Directional Feature Transformation
• Twin-GAN -- Unpaired Cross-Domain Image Translation with Weight-Sharing GANs
• Transformation-based Adversarial Video Prediction on Large-Scale Data
• ACE: Adapting to Changing Environments for Semantic Segmentation
• MISO: Mutual Information Loss with Stochastic Style Representations for Multimodal Image-to-Image Translation
• Mask-Guided Portrait Editing with Conditional GANs
• Learning Spatial Pyramid Attentive Pooling in Image Synthesis and Image-to-Image Translation
• Few-shot Video-to-Video Synthesis
• GANHopper: Multi-Hop GAN for Unsupervised Image-to-Image Translation
• GANwriting: Content-Conditioned Generation of Styled Handwritten Word Images
• Unsupervised Speech Domain Adaptation Based on Disentangled Representation Learning for Robust Speech Recognition
• Unpaired Pose Guided Human Image Generation
• SDIT: Scalable and Diverse Cross-domain Image Translation
• WarpGAN: Automatic Caricature Generation
• Beyond Domain Adaptation: Unseen Domain Encapsulation via Universal Non-volume Preserving Models
• Recent Advances in Fibrosis and Scar Segmentation from Cardiac MRI: A State-of-the-Art Review and Future Perspectives
• Generation of Synthetic Electronic Health Records Using a Federated GAN
• Continuous Conditional Generative Adversarial Networks (cGAN) with Generator Regularization
• Unsupervised Neural Sensor Models for Synthetic LiDAR Data Augmentation
• Deep Network Interpolation for Continuous Imagery Effect Transition
• Texture Deformation Based Generative Adversarial Networks for Face Editing
• Distilling portable Generative Adversarial Networks for Image Translation
• EFANet: Exchangeable Feature Alignment Network for Arbitrary Style Transfer
• Play as You Like: Timbre-enhanced Multi-modal Music Style Transfer
• Im2Pencil: Controllable Pencil Illustration from Photographs
• Learning Disentangled Representations of Satellite Image Time Series
• FaceShapeGene: A Disentangled Shape Representation for Flexible Face Image Editing
• Sparsely Grouped Multi-task Generative Adversarial Networks for Facial Attribute Manipulation
• Learning to Caricature via Semantic Shape Transform
• Unsupervised Image Super-Resolution with an Indirect Supervised Path
• Towards Lifelong Self-Supervision For Unpaired Image-to-Image Translation
• Attribute-Driven Spontaneous Motion in Unpaired Image Translation
• How to make a pizza: Learning a compositional layer-based GAN model
• ClimateGAN: Raising Climate Change Awareness by Generating Images of Floods
• Diverse Image Generation via Self-Conditioned GANs
• Kernel Mean Matching for Content Addressability of GANs
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• Memory-guided Unsupervised Image-to-image Translation
• Generative Creativity: Adversarial Learning for Bionic Design
• Inserting Videos into Videos
• What and Where to Translate: Local Mask-based Image-to-Image Translation
• Few-Shot Unsupervised Image-to-Image Translation on complex scenes
• Depth Guided Adaptive Meta-Fusion Network for Few-shot Video Recognition
• MCMI: Multi-Cycle Image Translation with Mutual Information Constraints
• Neural Wireframe Renderer: Learning Wireframe to Image Translations
• Bridging the Gap between Label- and Reference-based Synthesis in Multi-attribute Image-to-Image Translation
• Dual-MTGAN: Stochastic and Deterministic Motion Transfer for Image-to-Video Synthesis
• Learning Resolution-Invariant Deep Representations for Person Re-Identification
• Multi-component Image Translation for Deep Domain Generalization
• Delving into Rectifiers in Style-Based Image Translation
• Semi-Supervised Monaural Singing Voice Separation With a Masking Network Trained on Synthetic Mixtures
• LSC-GAN: Latent Style Code Modeling for Continuous Image-to-image Translation
• AE-StyleGAN: Improved Training of Style-Based Auto-Encoders
• BSD-GAN: Branched Generative Adversarial Network for Scale-Disentangled Representation Learning and Image Synthesis
• Synthesizing Photorealistic Images with Deep Generative Learning