Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input

Abstract

Real-time simultaneous tracking of hands manipulating and interacting with external objects has many potential applications in augmented reality, tangible computing, and wearable computing. However, due to difficult occlusions, fast motions, and uniform hand appearance, jointly tracking hand and object pose is more challenging than tracking either of the two separately. Many previous approaches resort to complex multi-camera setups to remedy the occlusion problem and often employ expensive segmentation and optimization steps which makes real-time tracking impossible. In this paper, we propose a real-time solution that uses a single commodity RGB-D camera. The core of our approach is a 3D articulated Gaussian mixture alignment strategy tailored to hand-object tracking that allows fast pose optimization. The alignment energy uses novel regularizers to address occlusions and hand-object contacts. For added robustness, we guide the optimization with discriminative part classification of the hand and segmentation of the object. We conducted extensive experiments on several existing datasets and introduce a new annotated hand-object dataset. Quantitative and qualitative results show the key advantages of our method: speed, accuracy, and robustness.

arXiv commentProceedings of ECCV 2016

References in corpus (3)

• Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
• Fast and Robust Hand Tracking Using Detection-Guided Optimization
• Capturing Hand Motion with an RGB-D Sensor, Fusing a Generative Model with Salient Points

Cited by in corpus (15)

• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• 3D Hand Shape and Pose Estimation from a Single RGB Image
• Monocular Total Capture: Posing Face, Body, and Hands in the Wild
• First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
• RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images
• A Technology-aided Multi-modal Training Approach to Assist Abdominal Palpation Training and its Assessment in Medical Education
• DenseAttentionSeg: Segment Hands from Interacted Objects Using Depth Input
• Simultaneous Hand Pose and Skeleton Bone-Lengths Estimation from a Single Depth Image
• GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB
• DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth
• Back to RGB: 3D tracking of hands and hand-object interactions based on short-baseline stereo
• Virtual Reality over Wireless Networks: Quality-of-Service Model and Learning-Based Resource Management
• Disentangling Pose from Appearance in Monochrome Hand Images
• MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation
• HANDS18: Methods, Techniques and Applications for Hand Observation