Memory-guided Unsupervised Image-to-image Translation

Abstract

We present a novel unsupervised framework for instance-level image-to-image translation. Although recent advances have been made by incorporating additional object annotations, existing methods often fail to handle images with multiple disparate objects. The main cause is that, during inference, they apply a global style to the whole image and do not consider the large style discrepancy between instance and background, or within instances. To address this problem, we propose a class-aware memory network that explicitly reasons about local style variations. A key-values memory structure, with a set of read/update operations, is introduced to record class-wise style variations and access them without requiring an object detector at the test time. The key stores a domain-agnostic content representation for allocating memory items, while the values encode domain-specific style representations. We also present a feature contrastive loss to boost the discriminative power of memory items. We show that by incorporating our memory, we can transfer class-aware and accurate style representations across domains. Experimental results demonstrate that our model outperforms recent instance-level methods and achieves state-of-the-art performance.

CVPR 2021

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Conditional Generative Adversarial Nets
• Improved Techniques for Training GANs
• Toward Multimodal Image-to-Image Translation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• Ask Me Anything: Dynamic Memory Networks for Natural Language Processing
• Unsupervised Image-to-Image Translation Networks
• Unsupervised Cross-Domain Image Generation
• DualGAN: Unsupervised Dual Learning for Image-to-Image Translation
• Multimodal Unsupervised Image-to-Image Translation
• BAGAN: Data Augmentation with Balancing GAN
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• InstaGAN: Instance-aware Image-to-Image Translation
• Domain Adaptation for Object Detection via Style Consistency
• Diversify and Match: A Domain Adaptive Representation Learning Paradigm for Object Detection
• Learning Dynamic Memory Networks for Object Tracking
• Image to Image Translation for Domain Adaptation