JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image

Abstract

State-of-the-art single depth image-based 3D hand pose estimation methods are based on dense predictions, including voxel-to-voxel predictions, point-to-point regression, and pixel-wise estimations. Despite the good performance, those methods have a few issues in nature, such as the poor trade-off between accuracy and efficiency, and plain feature representation learning with local convolutions. In this paper, a novel pixel-wise prediction-based method is proposed to address the above issues. The key ideas are two-fold: a) explicitly modeling the dependencies among joints and the relations between the pixels and the joints for better local feature representation learning; b) unifying the dense pixel-wise offset predictions and direct joint regression for end-to-end training. Specifically, we first propose a graph convolutional network (GCN) based joint graph reasoning module to model the complex dependencies among joints and augment the representation capability of each pixel. Then we densely estimate all pixels' offsets to joints in both image plane and depth space and calculate the joints' positions by a weighted average over all pixels' predictions, totally discarding the complex postprocessing operations. The proposed model is implemented with an efficient 2D fully convolutional network (FCN) backbone and has only about 1.4M parameters. Extensive experiments on multiple 3D hand pose estimation benchmarks demonstrate that the proposed method achieves new state-of-the-art accuracy while running very efficiently with around a speed of 110fps on a single NVIDIA 1080Ti GPU.

Accepted by ECCV2020 as a Spotlight paper

References in corpus (10)

• Semi-Supervised Classification with Graph Convolutional Networks
• Hands Deep in Deep Learning for Hand Pose Estimation
• Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Pose Guided Structured Region Ensemble Network for Cascaded Hand Pose Estimation
• Model-based Deep Hand Pose Estimation
• The 2017 Hands in the Million Challenge on 3D Hand Pose Estimation
• Towards Good Practices for Deep 3D Hand Pose Estimation
• End-to-end Global to Local CNN Learning for Hand Pose Recovery in Depth Data
• V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map