MediaPipe Hands: On-device Real-time Hand Tracking

Abstract

We present a real-time on-device hand tracking pipeline that predicts hand skeleton from single RGB camera for AR/VR applications. The pipeline consists of two models: 1) a palm detector, 2) a hand landmark model. It's implemented via MediaPipe, a framework for building cross-platform ML solutions. The proposed model and pipeline architecture demonstrates real-time inference speed on mobile GPUs and high prediction quality. MediaPipe Hands is open sourced at https://mediapipe.dev.

5 pages, 7 figures; CVPR Workshop on Computer Vision for Augmented and Virtual Reality, Seattle, WA, USA, 2020

References in corpus (2)

• BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs
• Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs

Cited by in corpus (16)

• Human keypoint detection for close proximity human-robot interaction
• Thermodynamics-informed neural networks for physically realistic mixed reality
• HanDiffuser: Text-to-Image Generation With Realistic Hand Appearances
• FollowMe: a Robust Person Following Framework Based on Re-Identification and Gestures
• Workspace Optimization Techniques to Improve Prediction of Human Motion During Human-Robot Collaboration
• MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing
• CCOM-HuQin: an Annotated Multimodal Chinese Fiddle Performance Dataset
• QUB-PHEO: A Visual-Based Dyadic Multi-View Dataset for Intention Inference in Collaborative Assembly
• Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective
• Long-Distance Gesture Recognition using Dynamic Neural Networks
• Beyond-Voice: Towards Continuous 3D Hand Pose Tracking on Commercial Home Assistant Devices
• Testing Human-Hand Segmentation on In-Distribution and Out-of-Distribution Data in Human-Robot Interactions Using a Deep Ensemble Model
• MapIO: Embodied Interaction for the Accessibility of Tactile Maps Through Augmented Touch Exploration and Conversation
• Pen Spinning Hand Movement Analysis Using MediaPipe Hands
• Fast Monocular Hand Pose Estimation on Embedded Systems
• Single RGB-D Camera Teleoperation for General Robotic Manipulation