Multimodal Image-to-Image Translation via Mutual Information Estimation and Maximization

Abstract

Multimodal image-to-image translation (I2IT) aims to learn a conditional distribution that explores multiple possible images in the target domain given an input image in the source domain. Conditional generative adversarial networks (cGANs) are often adopted for modeling such a conditional distribution. However, cGANs are prone to ignore the latent code and learn a unimodal distribution in conditional image synthesis, which is also known as the mode collapse issue of GANs. To solve the problem, we propose a simple yet effective method that explicitly estimates and maximizes the mutual information between the latent code and the output image in cGANs by using a deep mutual information neural estimator in this paper. Maximizing the mutual information strengthens the statistical dependency between the latent code and the output image, which prevents the generator from ignoring the latent code and encourages cGANs to fully utilize the latent code for synthesizing diverse results. Our method not only provides a new perspective from information theory to improve diversity for I2IT but also achieves disentanglement between the source domain content and the target domain style for free.

References in corpus (19)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Self-Attention Generative Adversarial Networks
• Learning deep representations by mutual information estimation and maximization
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Equitability, mutual information, and the maximal information coefficient
• Adversarial Feature Learning
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Adversarially Learned Inference
• VEEGAN: Reducing Mode Collapse in GANs using Implicit Variational Learning
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Mode Regularized Generative Adversarial Networks
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• On GANs and GMMs
• Multi-mapping Image-to-Image Translation via Learning Disentanglement