TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets

Abstract

Image-to-image (I2I) translation has matured in recent years and is able to generate high-quality realistic images. However, despite current success, it still faces important challenges when applied to small domains. Existing methods use transfer learning for I2I translation, but they still require the learning of millions of parameters from scratch. This drawback severely limits its application on small domains. In this paper, we propose a new transfer learning for I2I translation (TransferI2I). We decouple our learning process into the image generation step and the I2I translation step. In the first step we propose two novel techniques: source-target initialization and self-initialization of the adaptor layer. The former finetunes the pretrained generative model (e.g., StyleGAN) on source and target data. The latter allows to initialize all non-pretrained network parameters without the need of any data. These techniques provide a better initialization for the I2I translation step. In addition, we introduce an auxiliary GAN that further facilitates the training of deep I2I systems even from small datasets. In extensive experiments on three datasets, (Animal faces, Birds, and Foods), we show that we outperform existing methods and that mFID improves on several datasets with over 25 points.

arXiv commentTechnical report

References in corpus (16)

• Overcoming catastrophic forgetting in neural networks
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Toward Multimodal Image-to-Image Translation
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Invertible Conditional GANs for image editing
• U-GAT-IT: Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Unsupervised Attention-guided Image to Image Translation
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Freeze the Discriminator: a Simple Baseline for Fine-Tuning GANs
• A Unified Feature Disentangler for Multi-Domain Image Translation and Manipulation
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Multi-mapping Image-to-Image Translation via Learning Disentanglement
• Few-shot Image Generation with Elastic Weight Consolidation
• Structural-analogy from a Single Image Pair
• DeepI2I: Enabling Deep Hierarchical Image-to-Image Translation by Transferring from GANs