Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera

Abstract

We present a novel method for real-time pose and shape reconstruction of two strongly interacting hands. Our approach is the first two-hand tracking solution that combines an extensive list of favorable properties, namely it is marker-less, uses a single consumer-level depth camera, runs in real time, handles inter- and intra-hand collisions, and automatically adjusts to the user's hand shape. In order to achieve this, we embed a recent parametric hand pose and shape model and a dense correspondence predictor based on a deep neural network into a suitable energy minimization framework. For training the correspondence prediction network, we synthesize a two-hand dataset based on physical simulations that includes both hand pose and shape annotations while at the same time avoiding inter-hand penetrations. To achieve real-time rates, we phrase the model fitting in terms of a nonlinear least-squares problem so that the energy can be optimized based on a highly efficient GPU-based Gauss-Newton optimizer. We show state-of-the-art results in scenes that exceed the complexity level demonstrated by previous work, including tight two-hand grasps, significant inter-hand occlusions, and gesture interaction.

arXiv commentACM Transactions on Graphics (Proceedings SIGGRAPH 2019)

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Embodied Hands: Modeling and Capturing Hands and Bodies Together
• Capturing Hands in Action using Discriminative Salient Points and Physics Simulation
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• Fast and Robust Hand Tracking Using Detection-Guided Optimization
• Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
• Real-Time Hand Tracking Using a Sum of Anisotropic Gaussians Model
• 3D Hand Pose Detection in Egocentric RGB-D Images

Cited by in corpus (13)

• Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera
• TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric Vision
• Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection
• Pyramid Deep Fusion Network for Two-Hand Reconstruction from RGB-D Images
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction
• EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
• Two-hand Global 3D Pose Estimation Using Monocular RGB
• The Phong Surface: Efficient 3D Model Fitting using Lifted Optimization
• Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras
• Wrist2Finger: Sensing Fingertip Force for Force-Aware Hand Interaction with a Ring-Watch Wearable
• Monocular 3D Reconstruction of Interacting Hands via Collision-Aware Factorized Refinements
• HaDR: Applying Domain Randomization for Generating Synthetic Multimodal Dataset for Hand Instance Segmentation in Cluttered Industrial Environments
• Parallel mesh reconstruction streams for pose estimation of interacting hands