MonoCap: Monocular Human Motion Capture using a CNN Coupled with a Geometric Prior

Abstract

Recovering 3D full-body human pose is a challenging problem with many applications. It has been successfully addressed by motion capture systems with body worn markers and multiple cameras. In this paper, we address the more challenging case of not only using a single camera but also not leveraging markers: going directly from 2D appearance to 3D geometry. Deep learning approaches have shown remarkable abilities to discriminatively learn 2D appearance features. The missing piece is how to integrate 2D, 3D and temporal information to recover 3D geometry and account for the uncertainties arising from the discriminative model. We introduce a novel approach that treats 2D joint locations as latent variables whose uncertainty distributions are given by a deep fully convolutional neural network. The unknown 3D poses are modeled by a sparse representation and the 3D parameter estimates are realized via an Expectation-Maximization algorithm, where it is shown that the 2D joint location uncertainties can be conveniently marginalized out during inference. Extensive evaluation on benchmark datasets shows that the proposed approach achieves greater accuracy over state-of-the-art baselines. Notably, the proposed approach does not require synchronized 2D-3D data for training and is applicable to "in-the-wild" images, which is demonstrated with the MPII dataset.

arXiv commentAccepted by PAMI. Extended version of the following paper: Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video. X Zhou, M Zhu, S Leonardos, K Derpanis, K Daniilidis. CVPR 2016. arXiv admin note: substantial text overlap with arXiv:1511.09439

References in corpus (7)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Fully Convolutional Networks for Semantic Segmentation
• Manopt, a Matlab toolbox for optimization on manifolds
• Learning Human Pose Estimation Features with Convolutional Networks
• Sparse Representation for 3D Shape Estimation: A Convex Relaxation Approach
• Flowing ConvNets for Human Pose Estimation in Videos

Cited by in corpus (8)

• Self-supervised Learning of Motion Capture
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• OriNet: A Fully Convolutional Network for 3D Human Pose Estimation
• Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation
• Integral Human Pose Regression
• DRPose3D: Depth Ranking in 3D Human Pose Estimation
• Ordinal Depth Supervision for 3D Human Pose Estimation
• Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose