Learning to Estimate 3D Human Pose and Shape from a Single Color Image

Abstract

This work addresses the problem of estimating the full body 3D human pose and shape from a single color image. This is a task where iterative optimization-based solutions have typically prevailed, while Convolutional Networks (ConvNets) have suffered because of the lack of training data and their low resolution 3D predictions. Our work aims to bridge this gap and proposes an efficient and effective direct prediction method based on ConvNets. Central part to our approach is the incorporation of a parametric statistical body shape model (SMPL) within our end-to-end framework. This allows us to get very detailed 3D mesh results, while requiring estimation only of a small number of parameters, making it friendly for direct network prediction. Interestingly, we demonstrate that these parameters can be predicted reliably only from 2D keypoints and masks. These are typical outputs of generic 2D human analysis ConvNets, allowing us to relax the massive requirement that images with 3D shape ground truth are available for training. Simultaneously, by maintaining differentiability, at training time we generate the 3D mesh from the estimated parameters and optimize explicitly for the surface using a 3D per-vertex loss. Finally, a differentiable renderer is employed to project the 3D mesh to the image, which enables further refinement of the network, by optimizing for the consistency of the projection with 2D annotations (i.e., 2D keypoints or masks). The proposed approach outperforms previous baselines on this task and offers an attractive solution for direct prediction of 3D shape from a single color image.

arXiv commentCVPR 2018 Camera Ready

References in corpus (21)

• Learning from Synthetic Humans
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Stacked Hourglass Networks for Human Pose Estimation
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach
• A compact formula for the derivative of a 3-D rotation in exponential coordinates
• A simple yet effective baseline for 3d human pose estimation
• Flowing ConvNets for Human Pose Estimation in Videos
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• Towards Accurate Markerless Human Shape and Pose Estimation over Time
• Compositional Human Pose Regression
• Structured Prediction of 3D Human Pose with Deep Neural Networks
• Unite the People: Closing the Loop Between 3D and 2D Human Representations
• MonoPerfCap: Human Performance Capture from Monocular Video
• Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
• DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation
• General Automatic Human Shape and Motion Capture Using Volumetric Contour Cues
• Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs
• Monocular 3D Human Pose Estimation In The Wild Using Improved CNN Supervision
• Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation
• 3D Human Pose Estimation from a Single Image via Distance Matrix Regression

Cited by in corpus (8)

• Fast and Robust Multi-Person 3D Pose Estimation from Multiple Views
• Single Image 3D Hand Reconstruction with Mesh Convolutions
• ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving
• Shape-from-Mask: A Deep Learning Based Human Body Shape Reconstruction from Binary Mask Images
• Re-Identification Supervised Texture Generation
• Fashion is Taking Shape: Understanding Clothing Preference Based on Body Shape From Online Sources
• Single-Shot Multi-Person 3D Pose Estimation From Monocular RGB
• A Novel Self-Intersection Penalty Term for Statistical Body Shape Models and Its Applications in 3D Pose Estimation