Mix and match networks: encoder-decoder alignment for zero-pair image translation
arXiv:1804.02199
Abstract
We address the problem of image translation between domains or modalities for which no direct paired data is available (i.e. zero-pair translation). We propose mix and match networks, based on multiple encoders and decoders aligned in such a way that other encoder-decoder pairs can be composed at test time to perform unseen image translation tasks between domains or modalities for which explicit paired samples were not seen during training. We study the impact of autoencoders, side information and losses in improving the alignment and transferability of trained pairwise translation models to unseen translations. We show our approach is scalable and can perform colorization and style transfer between unseen combinations of domains. We evaluate our system in a challenging cross-modal setting where semantic segmentation is estimated from depth images, without explicit access to any depth-semantic segmentation training pairs. Our model outperforms baselines based on pix2pix and CycleGAN models.
arXiv commentAccepted CVPR 2018
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
- Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections
- Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
- Unsupervised Cross-Domain Image Generation
- Least Squares Generative Adversarial Networks
- A Teacher-Student Framework for Zero-Resource Neural Machine Translation
- Recent Advances in Zero-shot Recognition