GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction

Abstract

In the past few years, a lot of work has been done towards reconstructing the 3D facial structure from single images by capitalizing on the power of Deep Convolutional Neural Networks (DCNNs). In the most recent works, differentiable renderers were employed in order to learn the relationship between the facial identity features and the parameters of a 3D morphable model for shape and texture. The texture features either correspond to components of a linear texture space or are learned by auto-encoders directly from in-the-wild images. In all cases, the quality of the facial texture reconstruction of the state-of-the-art methods is still not capable of modeling textures in high fidelity. In this paper, we take a radically different approach and harness the power of Generative Adversarial Networks (GANs) and DCNNs in order to reconstruct the facial texture and shape from single images. That is, we utilize GANs to train a very powerful generator of facial texture in UV space. Then, we revisit the original 3D Morphable Models (3DMMs) fitting approaches making use of non-linear optimization to find the optimal latent parameters that best reconstruct the test image but under a new perspective. We optimize the parameters with the supervision of pretrained deep identity features through our end-to-end differentiable framework. We demonstrate excellent results in photorealistic and identity preserving 3D face reconstructions and achieve for the first time, to the best of our knowledge, facial texture reconstruction with high-frequency details.

arXiv commentCVPR 2019 camera ready; Check project page: https://github.com/barisgecer/ganfit for full resolution results and more

References in corpus (4)

• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Deep Appearance Models for Face Rendering
• 3D Face Reconstruction with Geometry Details from a Single Image
• Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model

Cited by in corpus (20)

• SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction
• Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction
• AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs
• Self-supervised Learning of Interpretable Keypoints from Unlabelled Videos
• OSTeC: One-Shot Texture Completion
• A Sparse and Locally Coherent Morphable Face Model for Dense Semantic Correspondence Across Heterogeneous 3D Faces
• A Dataset and Benchmark Towards Multi-Modal Face Anti-Spoofing Under Surveillance Scenarios
• Adaptive Joint Optimization for 3D Reconstruction with Differentiable Rendering
• Neural Parameterization for Dynamic Human Head Editing
• Text to Image Synthesis using Stacked Conditional Variational Autoencoders and Conditional Generative Adversarial Networks
• Learning a Generalized Physical Face Model From Data
• Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions
• Modeling Caricature Expressions by 3D Blendshape and Dynamic Texture
• High-Quality 3D Face Reconstruction with Affine Convolutional Networks
• Implicit Shape and Appearance Priors for Few-Shot Full Head Reconstruction
• REFA: Real-time Egocentric Facial Animations for Virtual Reality
• Face Inverse Rendering via Hierarchical Decoupling
• Bringing Diversity from Diffusion Models to Semantic-Guided Face Asset Generation
• High-Quality Facial Albedo Generation for 3D Face Reconstruction from a Single Image using a Coarse-to-Fine Approach