Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose

Abstract

This paper addresses the challenge of 3D human pose estimation from a single color image. Despite the general success of the end-to-end learning paradigm, top performing approaches employ a two-step solution consisting of a Convolutional Network (ConvNet) for 2D joint localization and a subsequent optimization step to recover 3D pose. In this paper, we identify the representation of 3D pose as a critical issue with current ConvNet approaches and make two important contributions towards validating the value of end-to-end learning for this task. First, we propose a fine discretization of the 3D space around the subject and train a ConvNet to predict per voxel likelihoods for each joint. This creates a natural representation for 3D pose and greatly improves performance over the direct regression of joint coordinates. Second, to further improve upon initial estimates, we employ a coarse-to-fine prediction scheme. This step addresses the large dimensionality increase and enables iterative refinement and repeated processing of the image features. The proposed approach outperforms all state-of-the-art methods on standard benchmarks achieving a relative error reduction greater than 30% on average. Additionally, we investigate using our volumetric representation in a related architecture which is suboptimal compared to our end-to-end approach, but is of practical interest, since it enables training when no image with corresponding 3D groundtruth is available, and allows us to present compelling results for in-the-wild images.

arXiv commentCVPR 2017 Camera Ready. Project Page: https://www.seas.upenn.edu/~pavlakos/projects/volumetric/

References in corpus (9)

• Going Deeper with Convolutions
• Stacked Hourglass Networks for Human Pose Estimation
• Convolutional Pose Machines
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Flowing ConvNets for Human Pose Estimation in Videos
• Synthesizing Training Images for Boosting Human 3D Pose Estimation
• Structured Prediction of 3D Human Pose with Deep Neural Networks
• Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
• 3D Shape Estimation from 2D Landmarks: A Convex Relaxation Approach

Cited by in corpus (10)

• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach
• Large Pose 3D Face Reconstruction from a Single Image via Direct Volumetric CNN Regression
• MonoPerfCap: Human Performance Capture from Monocular Video
• Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation
• Integral Human Pose Regression
• Holistic Planimetric prediction to Local Volumetric prediction for 3D Human Pose Estimation
• An Integral Pose Regression System for the ECCV2018 PoseTrack Challenge
• An End-to-end Framework for Unconstrained Monocular 3D Hand Pose Estimation
• Explicit Spatiotemporal Joint Relation Learning for Tracking Human Pose