Toward Multimodal Image-to-Image Translation

Abstract

Many image-to-image translation problems are ambiguous, as a single input image may correspond to multiple possible outputs. In this work, we aim to model a \emph{distribution} of possible outputs in a conditional generative modeling setting. The ambiguity of the mapping is distilled in a low-dimensional latent vector, which can be randomly sampled at test time. A generator learns to map the given input, combined with this latent code, to the output. We explicitly encourage the connection between output and the latent code to be invertible. This helps prevent a many-to-one mapping from the latent code to the output during training, also known as the problem of mode collapse, and produces more diverse results. We explore several variants of this approach by employing different training objectives, network architectures, and methods of injecting the latent code. Our proposed method encourages bijective consistency between the latent encoding and output modes. We present a systematic comparison of our method and other variants on both perceptual realism and diversity.

arXiv commentNIPS 2017 Final paper. v4 updated acknowledgment. Website: https://junyanz.github.io/BicycleGAN/

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Generative Adversarial Text to Image Synthesis
• Conditional Image Generation with PixelCNN Decoders
• Pixel Recurrent Neural Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• One weird trick for parallelizing convolutional neural networks
• Energy-based Generative Adversarial Network
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• Adversarially Learned Inference
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Real-Time User-Guided Image Colorization with Learned Deep Priors
• TextureGAN: Controlling Deep Image Synthesis with Texture Patches
• PixelNN: Example-based Image Synthesis

Cited by in corpus (202)

• Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• Neo: A Learned Query Optimizer
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models
• Multimodal Unsupervised Image-to-Image Translation
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Stochastic Adversarial Video Prediction
• Disentangled Representation Learning in Cardiac Image Analysis
• VehicleNet: Learning Robust Visual Representation for Vehicle Re-identification
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• A Probabilistic U-Net for Segmentation of Ambiguous Images
• PiiGAN: Generative Adversarial Networks for Pluralistic Image Inpainting
• HiFaceGAN: Face Renovation via Collaborative Suppression and Replenishment
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Generative Adversarial Networks for Spatio-temporal Data: A Survey
• Data augmentation in microscopic images for material data mining
• Video-to-Video Synthesis
• Image to Images Translation for Multi-Task Organ Segmentation and Bone Suppression in Chest X-Ray Radiography
• CariGANs: Unpaired Photo-to-Caricature Translation
• Learning Disentangled Representations in the Imaging Domain
• Artistic Glyph Image Synthesis via One-Stage Few-Shot Learning
• Abnormal Colon Polyp Image Synthesis Using Conditional Adversarial Networks for Improved Detection Performance
• Image-to-image translation for cross-domain disentanglement
• Stochastic Segmentation Networks: Modelling Spatially Correlated Aleatoric Uncertainty
• Few-Shot Unsupervised Image-to-Image Translation
• Bi-Modality Medical Image Synthesis Using Semi-Supervised Sequential Generative Adversarial Networks
• UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models
• Bridging the Gap Between Computational Photography and Visual Recognition
• FrankenGAN: Guided Detail Synthesis for Building Mass-Models Using Style-Synchonized GANs
• TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• StructureFlow: Image Inpainting via Structure-aware Appearance Flow
• Imitating Latent Policies from Observation
• Multimodal 4DVarNets for the reconstruction of sea surface dynamics from SST-SSH synergies
• Facial-Sketch Synthesis: A New Challenge
• Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving
• Learning Beyond Human Expertise with Generative Models for Dental Restorations
• Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs
• Simultaneous segmentation and classification of the retinal arteries and veins from color fundus images
• 3D-Aware Scene Manipulation via Inverse Graphics
• VCGAN: Video Colorization with Hybrid Generative Adversarial Network
• Low-field magnetic resonance image enhancement via stochastic image quality transfer
• On the Detection of Digital Face Manipulation
• Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation
• Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
• Exemplar Guided Unsupervised Image-to-Image Translation with Semantic Consistency
• SmartSeed: Smart Seed Generation for Efficient Fuzzing
• Simple and Effective VAE Training with Calibrated Decoders
• InversionNet3D: Efficient and Scalable Learning for 3D Full Waveform Inversion
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• Spatially Controllable Image Synthesis with Internal Representation Collaging
• On the Evaluation of Conditional GANs
• Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks
• Self-Ensembling with GAN-based Data Augmentation for Domain Adaptation in Semantic Segmentation
• Density-aware Haze Image Synthesis by Self-Supervised Content-Style Disentanglement
• Text as Neural Operator: Image Manipulation by Text Instruction
• Physics-informed GANs for Coastal Flood Visualization
• ELEGANT: Exchanging Latent Encodings with GAN for Transferring Multiple Face Attributes
• Robust Conditional Generative Adversarial Networks
• AdvHaze: Adversarial Haze Attack
• Hierarchical Quantized Autoencoders
• A Unified Multi-Phase CT Synthesis and Classification Framework for Kidney Cancer Diagnosis with Incomplete Data
• StyleGAN2 Distillation for Feed-forward Image Manipulation
• MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks
• Privacy-Protection Drone Patrol System based on Face Anonymization
• GAN Prior Embedded Network for Blind Face Restoration in the Wild
• One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
• Structural-analogy from a Single Image Pair
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• Audio query-based music source separation
• Image-to-Image Translation: Methods and Applications
• COVID-19 CT Image Synthesis with a Conditional Generative Adversarial Network
• ReenactGAN: Learning to Reenact Faces via Boundary Transfer
• GMM-UNIT: Unsupervised Multi-Domain and Multi-Modal Image-to-Image Translation via Attribute Gaussian Mixture Modeling
• Geometry-Consistent Generative Adversarial Networks for One-Sided Unsupervised Domain Mapping
• Watermark Faker: Towards Forgery of Digital Image Watermarking
• Attention-Guided Generative Adversarial Networks for Unsupervised Image-to-Image Translation
• Gesture-to-Gesture Translation in the Wild via Category-Independent Conditional Maps
• Only a Matter of Style: Age Transformation Using a Style-Based Regression Model
• Deep Generative Models for Distribution-Preserving Lossy Compression
• Domain-knowledge Inspired Pseudo Supervision (DIPS) for Unsupervised Image-to-Image Translation Models to Support Cross-Domain Classification
• Example-Guided Style Consistent Image Synthesis from Semantic Labeling
• Content and Colour Distillation for Learning Image Translations with the Spatial Profile Loss
• DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control
• Deblurring by Realistic Blurring
• Example Forgetting: A Novel Approach to Explain and Interpret Deep Neural Networks in Seismic Interpretation
• Compositional Transformers for Scene Generation
• See Through Smoke: Robust Indoor Mapping with Low-cost mmWave Radar
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• Learning Fixed Points in Generative Adversarial Networks: From Image-to-Image Translation to Disease Detection and Localization
• TransGaGa: Geometry-Aware Unsupervised Image-to-Image Translation
• Multi-Mapping Image-to-Image Translation with Central Biasing Normalization
• Neural Design Network: Graphic Layout Generation with Constraints
• SingleGAN: Image-to-Image Translation by a Single-Generator Network using Multiple Generative Adversarial Learning
• Adversarial Image Composition with Auxiliary Illumination
• Few-shot Video-to-Video Synthesis
• Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Interactive Neural Painting
• Local Implicit Neural Representations for Multi-Sequence MRI Translation
• Six-channel Image Representation for Cross-domain Object Detection
• Scene Gated Social Graph: Pedestrian Trajectory Prediction Based on Dynamic Social Graphs and Scene Constraints
• CounteRGAN: Generating Realistic Counterfactuals with Residual Generative Adversarial Nets
• Learning Invariant Representation for Unsupervised Image Restoration
• GANHopper: Multi-Hop GAN for Unsupervised Image-to-Image Translation
• DUAL-GLOW: Conditional Flow-Based Generative Model for Modality Transfer
• Mocycle-GAN: Unpaired Video-to-Video Translation
• Diverse Image Synthesis from Semantic Layouts via Conditional IMLE
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• Reciprocal Translation between SAR and Optical Remote Sensing Images with Cascaded-Residual Adversarial Networks
• Sequential Attention GAN for Interactive Image Editing
• Image Generation from Layout
• GestureGAN for Hand Gesture-to-Gesture Translation in the Wild
• Exploring Explicit Domain Supervision for Latent Space Disentanglement in Unpaired Image-to-Image Translation
• Multimodal Shape Completion via Conditional Generative Adversarial Networks
• SDIT: Scalable and Diverse Cross-domain Image Translation
• A Mobile Robot Hand-Arm Teleoperation System by Vision and IMU
• Cooperative Training of Fast Thinking Initializer and Slow Thinking Solver for Conditional Learning
• Measuring the Biases and Effectiveness of Content-Style Disentanglement
• Asymmetric GANs for Image-to-Image Translation
• Mask Based Unsupervised Content Transfer
• Dual Contrastive Loss and Attention for GANs
• Pixel and Feature Level Based Domain Adaption for Object Detection in Autonomous Driving
• Towards Better Long-range Time Series Forecasting using Generative Adversarial Networks
• Variation-Aware Semantic Image Synthesis
• Half-body Portrait Relighting with Overcomplete Lighting Representation
• A multi-stage GAN for multi-organ chest X-ray image generation and segmentation
• Diversifying Semantic Image Synthesis and Editing via Class- and Layer-wise VAEs
• Model-Based Robust Deep Learning: Generalizing to Natural, Out-of-Distribution Data
• Semantically Multi-modal Image Synthesis
• Unpaired Image-to-Image Translation using Adversarial Consistency Loss
• Unsupervised Eyeglasses Removal in the Wild
• Copy the Old or Paint Anew? An Adversarial Framework for (non-) Parametric Image Stylization
• Resembled Generative Adversarial Networks: Two Domains with Similar Attributes
• Semi-supervised Learning for Few-shot Image-to-Image Translation
• Stochastic Conditional Generative Networks with Basis Decomposition
• Joint Learning of Generative Translator and Classifier for Visually Similar Classes
• Learning a Generative Transition Model for Uncertainty-Aware Robotic Manipulation
• Play as You Like: Timbre-enhanced Multi-modal Music Style Transfer
• Domain-Specific Mappings for Generative Adversarial Style Transfer
• Learning to Caricature via Semantic Shape Transform
• Animating Landscape: Self-Supervised Learning of Decoupled Motion and Appearance for Single-Image Video Synthesis
• Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches
• Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck
• Password-conditioned Anonymization and Deanonymization with Face Identity Transformers
• Sketch-to-Art: Synthesizing Stylized Art Images From Sketches
• Augmented Cyclic Consistency Regularization for Unpaired Image-to-Image Translation
• PD-GAN: Probabilistic Diverse GAN for Image Inpainting
• Rethinking conditional GAN training: An approach using geometrically structured latent manifolds
• Learning the Loss Functions in a Discriminative Space for Video Restoration
• Guided Image Inpainting: Replacing an Image Region by Pulling Content from Another Image
• Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
• MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing
• Integrating Categorical Semantics into Unsupervised Domain Translation
• Hierarchy Composition GAN for High-fidelity Image Synthesis
• Independent Encoder for Deep Hierarchical Unsupervised Image-to-Image Translation
• Structural Consistency and Controllability for Diverse Colorization
• Joint Semi-supervised 3D Super-Resolution and Segmentation with Mixed Adversarial Gaussian Domain Adaptation
• DLOW: Domain Flow for Adaptation and Generalization
• Conditional Generative Modeling via Learning the Latent Space
• Long-Term Human Video Generation of Multiple Futures Using Poses
• Adversarial sampling of unknown and high-dimensional conditional distributions
• Memory-guided Unsupervised Image-to-image Translation
• Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation
• Conditional Frechet Inception Distance
• DLGAN: Disentangling Label-Specific Fine-Grained Features for Image Manipulation
• A Layer-Based Sequential Framework for Scene Generation with GANs
• Unsupervised Representation Adversarial Learning Network: from Reconstruction to Generation
• Conditional Adversarial Camera Model Anonymization
• Semi-supervised mp-MRI Data Synthesis with StitchLayer and Auxiliary Distance Maximization
• PortraitGAN for Flexible Portrait Manipulation
• Unsupervised Many-to-Many Image-to-Image Translation Across Multiple Domains
• A Novel Framework for Image-to-image Translation and Image Compression
• What and Where to Translate: Local Mask-based Image-to-Image Translation
• TE141K: Artistic Text Benchmark for Text Effect Transfer
• Hierarchical Modes Exploring in Generative Adversarial Networks
• Deep CG2Real: Synthetic-to-Real Translation via Image Disentanglement
• Spectral Synthesis for Satellite-to-Satellite Translation
• Explorable Tone Mapping Operators
• Enhance Convolutional Neural Networks with Noise Incentive Block
• Sym-parameterized Dynamic Inference for Mixed-Domain Image Translation
• A Novel Technique for Evidence based Conditional Inference in Deep Neural Networks via Latent Feature Perturbation
• Unsupervised Image Transformation Learning via Generative Adversarial Networks
• Glioblastoma Multiforme Prognosis: MRI Missing Modality Generation, Segmentation and Radiogenomic Survival Prediction
• Style-Restricted GAN: Multi-Modal Translation with Style Restriction Using Generative Adversarial Networks
• Examining Performance of Sketch-to-Image Translation Models with Multiclass Automatically Generated Paired Training Data
• Conditional Invertible Neural Networks for Diverse Image-to-Image Translation
• TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets
• Image Manipulation with Perceptual Discriminators
• MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamics
• Variational Bayesian Framework for Advanced Image Generation with Domain-Related Variables
• Synthesizing Photorealistic Images with Deep Generative Learning
• Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis
• Unbiased Image Style Transfer
• Deep Learning-based Frozen Section to FFPE Translation
• Bridging the Gap between Label- and Reference-based Synthesis in Multi-attribute Image-to-Image Translation
• Seamless Satellite-image Synthesis
• MCMI: Multi-Cycle Image Translation with Mutual Information Constraints
• Image Generation and Recognition (Emotions)