Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images

Abstract

Though face rotation has achieved rapid progress in recent years, the lack of high-quality paired training data remains a great hurdle for existing methods. The current generative models heavily rely on datasets with multi-view images of the same person. Thus, their generated results are restricted by the scale and domain of the data source. To overcome these challenges, we propose a novel unsupervised framework that can synthesize photo-realistic rotated faces using only single-view image collections in the wild. Our key insight is that rotating faces in the 3D space back and forth, and re-rendering them to the 2D plane can serve as a strong self-supervision. We leverage the recent advances in 3D face modeling and high-resolution GAN to constitute our building blocks. Since the 3D rotation-and-render on faces can be applied to arbitrary angles without losing details, our approach is extremely suitable for in-the-wild scenarios (i.e. no paired data are available), where existing methods fall short. Extensive experiments demonstrate that our approach has superior synthesis quality as well as identity preservation over the state-of-the-art methods, across a wide range of poses and domains. Furthermore, we validate that our rotate-and-render framework naturally can act as an effective data augmentation engine for boosting modern face recognition systems even on strong baseline models.

arXiv commentTo appear in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. Code and models are available at: https://github.com/Hangz-nju-cuhk/Rotate-and-Render

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Learning Face Representation from Scratch
• Face Alignment Across Large Poses: A 3D Solution
• Face Alignment in Full Pose Range: A 3D Total Solution
• FD-GAN: Pose-guided Feature Distilling GAN for Robust Person Re-identification
• Image Inpainting via Generative Multi-column Convolutional Neural Networks
• Representation Learning by Rotating Your Faces
• Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis
• Learning a High Fidelity Pose Invariant Model for High-resolution Face Frontalization
• GANimation: Anatomically-aware Facial Animation from a Single Image
• Unsupervised Depth Estimation, 3D Face Rotation and Replacement
• Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images

Cited by in corpus (5)

• One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
• Rotate-and-Render: Unsupervised Photorealistic Face Rotation from Single-View Images
• Self-supervising Fine-grained Region Similarities for Large-scale Image Localization
• 3D Face Reconstruction for Forensic Recognition -- A Survey
• Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs