3D Hand Shape and Pose Estimation from a Single RGB Image

Abstract

This work addresses a novel and challenging problem of estimating the full 3D hand shape and pose from a single RGB image. Most current methods in 3D hand analysis from monocular RGB images only focus on estimating the 3D locations of hand keypoints, which cannot fully express the 3D shape of hand. In contrast, we propose a Graph Convolutional Neural Network (Graph CNN) based method to reconstruct a full 3D mesh of hand surface that contains richer information of both 3D hand shape and pose. To train networks with full supervision, we create a large-scale synthetic dataset containing both ground truth 3D meshes and 3D poses. When fine-tuning the networks on real-world datasets without 3D ground truth, we propose a weakly-supervised approach by leveraging the depth map as a weak supervision in training. Through extensive evaluations on our proposed new datasets and two public datasets, we show that our proposed method can produce accurate and reasonable 3D hand mesh, and can achieve superior 3D hand pose estimation accuracy when compared with state-of-the-art methods.

CVPR 2019 (Oral), project page: https://sites.google.com/site/geliuhaontu/home/cvpr2019

References in corpus (9)

• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
• 3D Hand Pose Tracking and Estimation Using Stereo Matching
• Robust 3D Hand Pose Estimation in Single Depth Images: from Single-View CNN to Multi-View CNNs
• Unite the People: Closing the Loop Between 3D and 2D Human Representations
• Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

Cited by in corpus (6)

• RGB2Hands: Real-Time Tracking of 3D Hand Interactions from Monocular RGB Video
• GraFormer: Graph Convolution Transformer for 3D Pose Estimation
• MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis
• Generative Model-Based Loss to the Rescue: A Method to Overcome Annotation Errors for Depth-Based Hand Pose Estimation
• GrabAR: Occlusion-aware Grabbing Virtual Objects in AR
• DGGAN: Depth-image Guided Generative Adversarial Networks for Disentangling RGB and Depth Images in 3D Hand Pose Estimation