Fine-grained Image-to-Image Transformation towards Visual Recognition

Abstract

Existing image-to-image transformation approaches primarily focus on synthesizing visually pleasing data. Generating images with correct identity labels is challenging yet much less explored. It is even more challenging to deal with image transformation tasks with large deformation in poses, viewpoints, or scales while preserving the identity, such as face rotation and object viewpoint morphing. In this paper, we aim at transforming an image with a fine-grained category to synthesize new images that preserve the identity of the input image, which can thereby benefit the subsequent fine-grained image recognition and few-shot learning tasks. The generated images, transformed with large geometric deformation, do not necessarily need to be of high visual quality but are required to maintain as much identity information as possible. To this end, we adopt a model based on generative adversarial networks to disentangle the identity related and unrelated factors of an image. In order to preserve the fine-grained contextual details of the input image during the deformable transformation, a constrained nonalignment connection method is proposed to construct learnable highways between intermediate convolution blocks in the generator. Moreover, an adaptive identity modulation mechanism is proposed to transfer the identity information into the output image effectively. Extensive experiments on the CompCars and Multi-PIE datasets demonstrate that our model preserves the identity of the generated images much better than the state-of-the-art image-to-image transformation models, and as a result significantly boosts the visual recognition performance in fine-grained few-shot learning.

arXiv commentCVPR 2020 Camera Ready version

References in corpus (28)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Deep Residual Learning for Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Improved Techniques for Training GANs
• Data Augmentation Generative Adversarial Networks
• Deformable Convolutional Networks
• Modulating early visual processing by language
• Multimodal Unsupervised Image-to-Image Translation
• Non-Local Recurrent Network for Image Restoration
• Generative Image Inpainting with Contextual Attention
• FiLM: Visual Reasoning with a General Conditioning Layer
• Non-local Neural Networks
• Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
• Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform
• On Self Modulation for Generative Adversarial Networks
• Low-Shot Learning from Imaginary Data
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• CR-GAN: Learning Complete Representations for Multi-view Generation
• Towards Open-Set Identity Preserving Face Synthesis
• Foreground-aware Image Inpainting
• Disentangled Person Image Generation
• CariGAN: Caricature Generation through Weakly Paired Adversarial Learning