CR-GAN: Learning Complete Representations for Multi-view Generation

Abstract

Generating multi-view images from a single-view input is an essential yet challenging problem. It has broad applications in vision, graphics, and robotics. Our study indicates that the widely-used generative adversarial network (GAN) may learn "incomplete" representations due to the single-pathway framework: an encoder-decoder network followed by a discriminator network. We propose CR-GAN to address this problem. In addition to the single reconstruction path, we introduce a generation sideway to maintain the completeness of the learned embedding space. The two learning pathways collaborate and compete in a parameter-sharing manner, yielding considerably improved generalization ability to "unseen" dataset. More importantly, the two-pathway framework makes it possible to combine both labeled and unlabeled data for self-supervised learning, which further enriches the embedding space for realistic generations. The experimental results prove that CR-GAN significantly outperforms state-of-the-art methods, especially when generating from "unseen" inputs in wild conditions.

arXiv comment7 pages, 9 figures, accepted by IJCAI 2018

References in corpus (7)

• Conditional Image Synthesis With Auxiliary Classifier GANs
• Improved Training of Wasserstein GANs
• Adversarially Learned Inference
• Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis
• Multi-View Image Generation from a Single-View
• A Recurrent Encoder-Decoder Network for Sequential Face Alignment
• Reconstruction-Based Disentanglement for Pose-invariant Face Recognition

Cited by in corpus (14)

• Generative deep learning as a tool for inverse design of high-entropy refractory alloys
• Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion
• Cartoonish sketch-based face editing in videos using identity deformation transfer
• SuperFront: From Low-resolution to High-resolution Frontal Face Synthesis
• Disentanglement for Discriminative Visual Recognition
• Template NeRF: Towards Modeling Dense Shape Correspondences from Category-Specific Object Images
• Learning Flow-based Feature Warping for Face Frontalization with Illumination Inconsistent Supervision
• Iterative Projection and Matching: Finding Structure-preserving Representatives and Its Application to Computer Vision
• Novel View Synthesis on Unpaired Data by Conditional Deformable Variational Auto-Encoder
• Exploring Biases and Prejudice of Facial Synthesis via Semantic Latent Space
• Pairwise-GAN: Pose-based View Synthesis through Pair-Wise Training
• Fine-grained Image-to-Image Transformation towards Visual Recognition
• ID-Unet: Iterative Soft and Hard Deformation for View Synthesis
• AE-StyleGAN: Improved Training of Style-Based Auto-Encoders