MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation

Abstract

Heatmap representations have formed the basis of human pose estimation systems for many years, and their extension to 3D has been a fruitful line of recent research. This includes 2.5D volumetric heatmaps, whose X and Y axes correspond to image space and Z to metric depth around the subject. To obtain metric-scale predictions, 2.5D methods need a separate post-processing step to resolve scale ambiguity. Further, they cannot localize body joints outside the image boundaries, leading to incomplete estimates for truncated images. To address these limitations, we propose metric-scale truncation-robust (MeTRo) volumetric heatmaps, whose dimensions are all defined in metric 3D space, instead of being aligned with image space. This reinterpretation of heatmap dimensions allows us to directly estimate complete, metric-scale poses without test-time knowledge of distance or relying on anthropometric heuristics, such as bone lengths. To further demonstrate the utility our representation, we present a differentiable combination of our 3D metric-scale heatmaps with 2D image-space ones to estimate absolute 3D pose (our MeTRAbs architecture). We find that supervision via absolute pose loss is crucial for accurate non-root-relative localization. Using a ResNet-50 backbone without further learned layers, we obtain state-of-the-art results on Human3.6M, MPI-INF-3DHP and MuPoTS-3D. Our code will be made publicly available to facilitate further research.

arXiv commentSee project page at https://vision.rwth-aachen.de/metrabs . Accepted for publication in the IEEE Transactions on Biometrics, Behavior, and Identity Science (TBIOM), Special Issue "Selected Best Works From Automated Face and Gesture Recognition 2020". Extended version of FG paper arXiv:2003.02953

References in corpus (10)

• DeepPose: Human Pose Estimation via Deep Neural Networks
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Learning from Synthetic Humans
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods
• Human Pose Regression by Combining Indirect Part Detection and Contextual Information
• Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition
• Single-shot 3D multi-person pose estimation in complex images
• Metric-Scale Truncation-Robust Heatmaps for 3D Human Pose Estimation
• What Face and Body Shapes Can Tell About Height

Cited by in corpus (2)

• Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputs
• One-Shot Open-Set Skeleton-Based Action Recognition