RGB-based 3D Hand Pose Estimation via Privileged Learning with Depth Images

Abstract

This paper proposes a method for hand pose estimation from RGB images that uses both external large-scale depth image datasets and paired depth and RGB images as privileged information at training time. We show that providing depth information during training significantly improves performance of pose estimation from RGB images during testing. We explore different ways of using this privileged information: (1) using depth data to initially train a depth-based network, (2) using the features from the depth-based network of the paired depth images to constrain mid-level RGB network weights, and (3) using the foreground mask, obtained from the depth data, to suppress the responses from the background area. By using paired RGB and depth images, we are able to supervise the RGB-based network to learn middle layer features that mimic that of the corresponding depth-based network, which is trained on large-scale, accurately annotated depth data. During testing, when only an RGB image is available, our method produces accurate 3D hand pose predictions. Our method is also tested on 2D hand pose estimation. Experiments on three public datasets show that the method outperforms the state-of-the-art methods for hand pose estimation using RGB image input.

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Do Deep Nets Really Need to be Deep?
• Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image
• Convolutional Pose Machines
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• 3D Hand Pose Tracking and Estimation Using Stereo Matching
• Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input
• Cross Modal Distillation for Supervision Transfer
• Training Group Orthogonal Neural Networks with Privileged Information

Cited by in corpus (2)

• SGM3D: Stereo Guided Monocular 3D Object Detection
• EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation