MM-Hand: 3D-Aware Multi-Modal Guided Hand Generative Network for 3D Hand Pose Synthesis

Abstract

Estimating the 3D hand pose from a monocular RGB image is important but challenging. A solution is training on large-scale RGB hand images with accurate 3D hand keypoint annotations. However, it is too expensive in practice. Instead, we have developed a learning-based approach to synthesize realistic, diverse, and 3D pose-preserving hand images under the guidance of 3D pose information. We propose a 3D-aware multi-modal guided hand generative network (MM-Hand), together with a novel geometry-based curriculum learning strategy. Our extensive experimental results demonstrate that the 3D-annotated images generated by MM-Hand qualitatively and quantitatively outperform existing options. Moreover, the augmented data can consistently improve the quantitative performance of the state-of-the-art 3D hand pose estimators on two benchmark datasets. The code will be available at https://github.com/ScottHoang/mm-hand.

Accepted by ACM Multimedia 2020

References in corpus (13)

• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor
• EnlightenGAN: Deep Light Enhancement without Paired Supervision
• 3D Hand Pose Tracking and Estimation Using Stereo Matching
• DeblurGAN-v2: Deblurring (Orders-of-Magnitude) Faster and Better
• Progressive Pose Attention Transfer for Person Image Generation
• 3D Hand Shape and Pose Estimation from a Single RGB Image
• Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis
• Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study
• Controllable Artistic Text Style Transfer via Shape-Matching GAN
• Unsupervised Person Image Generation with Semantic Parsing Transformation
• Unsupervised Person Image Synthesis in Arbitrary Poses
• Triple Generative Adversarial Networks

Cited by in corpus (4)

• Everybody Sign Now: Translating Spoken Language to Photo Realistic Sign Language Video
• Temporal-Aware Self-Supervised Learning for 3D Hand Pose and Mesh Estimation in Videos
• MVHM: A Large-Scale Multi-View Hand Mesh Benchmark for Accurate 3D Hand Pose Estimation
• Single Image 3D Object Estimation with Primitive Graph Networks