Contrastive Learning for Unpaired Image-to-Image Translation

Abstract

In image-to-image translation, each patch in the output should reflect the content of the corresponding patch in the input, independent of domain. We propose a straightforward method for doing so -- maximizing mutual information between the two, using a framework based on contrastive learning. The method encourages two elements (corresponding patches) to map to a similar point in a learned feature space, relative to other elements (other patches) in the dataset, referred to as negatives. We explore several critical design choices for making contrastive learning effective in the image synthesis setting. Notably, we use a multilayer, patch-based approach, rather than operate on entire images. Furthermore, we draw negatives from within the input image itself, rather than from the rest of the dataset. We demonstrate that our framework enables one-sided translation in the unpaired image-to-image translation setting, while improving quality and reducing training time. In addition, our method can even be extended to the training setting where each "domain" is only a single image.

arXiv commentECCV 2020. Please visit https://taesungp.github.io/ContrastiveUnpairedTranslation/ for introduction videos and more. v3 contains typo fixes and citation update

References in corpus (10)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Which Training Methods for GANs do actually Converge?
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Contrastive Multiview Coding
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Learning visual groups from co-occurrences in space and time
• InGAN: Capturing and Remapping the "DNA" of a Natural Image
• Harmonic Unpaired Image-to-image Translation

Cited by in corpus (13)

• Contrastive Representation Learning: A Framework and Review
• Generative Image Inpainting with Segmentation Confusion Adversarial Training and Contrastive Learning
• Contrastive Learning for Compact Single Image Dehazing
• Frequency-Controlled Diffusion Model for Versatile Text-Guided Image-to-Image Translation
• Total-Body Low-Dose CT Image Denoising using Prior Knowledge Transfer Technique with Contrastive Regularization Mechanism
• Inside Out: Transforming Images of Lab-Grown Plants for Machine Learning Applications in Agriculture
• Language-Driven Image Style Transfer
• Contrastive Unpaired Translation using Focal Loss for Patch Classification
• Transesophageal Echocardiography Generation using Anatomical Models
• Domain Adaptation of Networks for Camera Pose Estimation: Learning Camera Pose Estimation Without Pose Labels
• Visual Diver Face Recognition for Underwater Human-Robot Interaction
• Synthesizing Photorealistic Images with Deep Generative Learning
• Deep Learning-based Frozen Section to FFPE Translation