Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

Abstract

We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments. Existing methods typically fail for hand-object interactions in cluttered scenes imaged from egocentric viewpoints, common for virtual or augmented reality applications. Our approach uses two subsequently applied Convolutional Neural Networks (CNNs) to localize the hand and regress 3D joint locations. Hand localization is achieved by using a CNN to estimate the 2D position of the hand center in the input, even in the presence of clutter and occlusions. The localized hand position, together with the corresponding input depth value, is used to generate a normalized cropped image that is fed into a second CNN to regress relative 3D hand joint locations in real time. For added accuracy, robustness and temporal stability, we refine the pose estimates using a kinematic pose tracking energy. To train the CNNs, we introduce a new photorealistic dataset that uses a merged reality approach to capture and synthesize large amounts of annotated data of natural hand interaction in cluttered scenes. Through quantitative and qualitative evaluation, we show that our method is robust to self-occlusion and occlusions by objects, particularly in moving egocentric perspectives.

Accepted at the International Conference on Computer Vision (ICCV) 2017

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Unsupervised Domain Adaptation by Backpropagation
• Highway and Residual Networks learn Unrolled Iterative Estimation
• 3D Object Reconstruction from Hand-Object Interactions
• 3D Hand Pose Detection in Egocentric RGB-D Images
• Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
• Training a Feedback Loop for Hand Pose Estimation

Cited by in corpus (47)

• Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
• Analysis of the hands in egocentric vision: A survey
• 3D Hand Shape and Pose Estimation from a Single RGB Image
• Hand Pose Estimation: A Survey
• Survey on Hand Gesture Recognition from Visual Input
• Context-Aware Deep Spatio-Temporal Network for Hand Pose Estimation from Depth Images
• First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations
• HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation
• HandTailor: Towards High-Precision Monocular 3D Hand Recovery
• RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images
• Grasping Field: Learning Implicit Representations for Human Grasps
• DenseAttentionSeg: Segment Hands from Interacted Objects Using Depth Input
• Occlusion-aware Hand Pose Estimation Using Hierarchical Mixture Density Network
• Monocular Real-time Hand Shape and Motion Capture using Multi-modal Data
• GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB
• Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images
• MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis
• DeepHPS: End-to-end Estimation of 3D Hand Pose and Shape by Learning from Synthetic Depth
• Rotation-invariant Mixed Graphical Model Network for 2D Hand Pose Estimation
• Monocular Real-time Full Body Capture with Inter-part Correlations
• Measuring Generalisation to Unseen Viewpoints, Articulations, Shapes and Objects for 3D Hand Pose Estimation under Hand-Object Interaction
• Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation
• Visualizing the Invisible: Occluded Vehicle Segmentation and Recovery
• Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals
• JGR-P2O: Joint Graph Reasoning based Pixel-to-Offset Prediction Network for 3D Hand Pose Estimation from a Single Depth Image
• HUMBI: A Large Multiview Dataset of Human Body Expressions
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction
• Whole-Body Human Pose Estimation in the Wild
• Two-hand Global 3D Pose Estimation Using Monocular RGB
• SeqHAND:RGB-Sequence-Based 3D Hand Pose and Shape Estimation
• Leveraging Photometric Consistency over Time for Sparsely Supervised Hand-Object Reconstruction
• Disentangling Pose from Appearance in Monochrome Hand Images
• Semi-Supervised 3D Hand-Object Poses Estimation with Interactions in Time
• Serial-parallel Multi-Scale Feature Fusion for Anatomy-Oriented Hand Joint Detection
• Understanding Egocentric Hand-Object Interactions from Hand Pose Estimation
• MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation
• Adaptive Wasserstein Hourglass for Weakly Supervised Hand Pose Estimation from Monocular RGB
• GrabAR: Occlusion-aware Grabbing Virtual Objects in AR
• Egocentric 6-DoF Tracking of Small Handheld Objects
• Is First Person Vision Challenging for Object Tracking?
• Whose hand is this? Person Identification from Egocentric Hand Gestures
• EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
• Egocentric Hand-object Interaction Detection and Application
• Efficient 2.5D Hand Pose Estimation via Auxiliary Multi-Task Training for Embedded Devices
• Normalized Diversification
• HANDS18: Methods, Techniques and Applications for Hand Observation
• Human-Exoskeleton Kinematic Calibration to Improve Hand Tracking for Dexterous Teleoperation