3D Morphable Models as Spatial Transformer Networks

Abstract

In this paper, we show how a 3D Morphable Model (i.e. a statistical model of the 3D shape of a class of objects such as faces) can be used to spatially transform input data as a module (a 3DMM-STN) within a convolutional neural network. This is an extension of the original spatial transformer network in that we are able to interpret and normalise 3D pose changes and self-occlusions. The trained localisation part of the network is independently useful since it learns to fit a 3D morphable model to a single image. We show that the localiser can be trained using only simple geometric loss functions on a relatively small dataset yet is able to perform robust normalisation on highly uncontrolled images including occlusion, self-occlusion and large pose changes.

Accepted to ICCV 2017 2nd Workshop on Geometry Meets Deep Learning

References in corpus (3)

• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression
• InverseFaceNet: Deep Monocular Inverse Face Rendering

Cited by in corpus (16)

• Self-supervised learning of a facial attribute embedding from video
• 3D Face Reconstruction from A Single Image Assisted by 2D Face Images in the Wild
• Joint 3D Face Reconstruction and Dense Alignment with Position Map Regression Network
• Mesh Guided One-shot Face Reenactment using Graph Convolutional Networks
• Accurate 3D Face Reconstruction with Weakly-Supervised Learning: From Single Image to Image Set
• X2Face: A network for controlling face generation by using images, audio, and pose codes
• Beyond 3DMM: Learning to Capture High-fidelity 3D Face Shape
• Weakly-Supervised Multi-Face 3D Reconstruction
• Synthesizing facial photometries and corresponding geometries using generative adversarial networks
• Infinite 3D Landmarks: Improving Continuous 2D Facial Landmark Detection
• Deep Positional and Relational Feature Learning for Rotation-Invariant Point Cloud Analysis
• Vision-Language Modeling with Regularized Spatial Transformer Networks for All Weather Crosswind Landing of Aircraft
• A Review of 3D Face Reconstruction From a Single Image
• Statistical transformer networks: learning shape and appearance models via self supervision
• Convolutional Point-set Representation: A Convolutional Bridge Between a Densely Annotated Image and 3D Face Alignment
• Robust Attacks on Deep Learning Face Recognition in the Physical World