Mix and match networks: cross-modal alignment for zero-pair image-to-image translation
arXiv:1903.04294
Abstract
This paper addresses the problem of inferring unseen cross-modal image-to-image translations between multiple modalities. We assume that only some of the pairwise translations have been seen (i.e. trained) and infer the remaining unseen translations (where training pairs are not available). We propose mix and match networks, an approach where multiple encoders and decoders are aligned in such a way that the desired translation can be obtained by simply cascading the source encoder and the target decoder, even when they have not interacted during the training stage (i.e. unseen). The main challenge lies in the alignment of the latent representations at the bottlenecks of encoder-decoder pairs. We propose an architecture with several tools to encourage alignment, including autoencoders and robust side information and latent consistency losses. We show the benefits of our approach in terms of effectiveness and scalability compared with other pairwise image-to-image translation approaches. We also propose zero-pair cross-modal image translation, a challenging setting where the objective is inferring semantic segmentation from depth (and vice-versa) without explicit segmentation-depth pairs, and only from two (disjoint) segmentation-RGB and depth-RGB training sets. We observe that a certain part of the shared information between unseen modalities might not be reachable, so we further propose a variant that leverages pseudo-pairs which allows us to exploit this shared information between the unseen modalities.
arXiv commentAccepted by IJCV
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Conditional Generative Adversarial Nets
- Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
- Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
- Invertible Conditional GANs for image editing
- Disentangling factors of variation in deep representations using adversarial training
- Zero Shot Recognition with Unreliable Attributes
- Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs
- Recent Advances in Zero-shot Recognition