On Learning 3D Face Morphable Model from In-the-wild Images

Abstract

As a classic statistical model of 3D facial shape and albedo, 3D Morphable Model (3DMM) is widely used in facial analysis, e.g., model fitting, image synthesis. Conventional 3DMM is learned from a set of 3D face scans with associated well-controlled 2D face images, and represented by two sets of PCA basis functions. Due to the type and amount of training data, as well as, the linear bases, the representation power of 3DMM can be limited. To address these problems, this paper proposes an innovative framework to learn a nonlinear 3DMM model from a large set of in-the-wild face images, without collecting 3D face scans. Specifically, given a face image as input, a network encoder estimates the projection, lighting, shape and albedo parameters. Two decoders serve as the nonlinear 3DMM to map from the shape and albedo parameters to the 3D shape and albedo, respectively. With the projection parameter, lighting, 3D shape, and albedo, a novel analytically-differentiable rendering layer is designed to reconstruct the original input face. The entire network is end-to-end trainable with only weak supervision. We demonstrate the superior representation power of our nonlinear 3DMM over its linear counterpart, and its contribution to face alignment, 3D reconstruction, and face editing.

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). Conference version: arXiv:1804.03786 (CVPR'18). Source code: https://github.com/tranluan/Nonlinear_Face_3DMM , Project webpage: http://cvlab.cse.msu.edu/project-nonlinear-3dmm.html

References in corpus (4)

• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• Face Alignment in Full Pose Range: A 3D Total Solution
• Representation Learning by Rotating Your Faces
• Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis

Cited by in corpus (26)

• RetinaFace: Single-stage Dense Face Localisation in the Wild
• Representation Learning by Rotating Your Faces
• SADRNet: Self-Aligned Dual Face Regression Networks for Robust 3D Dense Face Alignment and Reconstruction
• Fast-GANFIT: Generative Adversarial Network for High Fidelity 3D Face Reconstruction
• AvatarMe++: Facial Shape and BRDF Inference with Photorealistic Rendering-Aware GANs
• OSTeC: One-Shot Texture Completion
• On the Detection of Digital Face Manipulation
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape
• Towards High-fidelity Nonlinear 3D Face Morphable Model
• Single-Shot Implicit Morphable Faces with Consistent Texture Parameterization
• DOVE: Learning Deformable 3D Objects by Watching Videos
• On Learning Disentangled Representations for Gait Recognition
• FaceScape: a Large-scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction
• Physics-Guided Spoof Trace Disentanglement for Generic Face Anti-Spoofing
• VRMM: A Volumetric Relightable Morphable Head Model
• Towards Interpretable Face Recognition
• Make a Face: Towards Arbitrary High Fidelity Face Manipulation
• Dense 3D Face Decoding over 2500FPS: Joint Texture & Shape Convolutional Mesh Decoders
• StyleUV: Diverse and High-fidelity UV Map Generative Model
• Learning Formation of Physically-Based Face Attributes
• Riggable 3D Face Reconstruction via In-Network Optimization
• Face Inverse Rendering via Hierarchical Decoupling
• A Review of 3D Face Reconstruction From a Single Image
• High-fidelity Face Tracking for AR/VR via Deep Lighting Adaptation
• Pixel Sampling for Style Preserving Face Pose Editing
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction