mmHand: Toward Pixel-Level-Accuracy Hand Localization Using a Single Commodity mmWave Device

Abstract

The hand localization problem has been a longstanding focus due to its many applications. The task involves modeling the hand as a singular point and determining its position within a defined coordinate system. However, due to data modality limitations, existing hand localization technologies face several challenges. For example, vision-based localization raises privacy concerns, while wearable-based methods compromise user comfort. In this paper, we introduce mmHand, a new device-free, privacy-preserving dynamic hand localization system with pixel-level accuracy, using a single commodity mmWave device. We first propose a mmImage generation tool to fully extract spatial information from raw mmWave data and introduce a novel 2D image-format representation of mmWave data. Next, we design a framework that provides a new quality evaluation method and pixel space labeling for the mmWave data. Finally, we present a cross-modality spatial feature-enhanced model with high spatial feature extraction capabilities, which can accurately localize hand positions at the pixel level in the mmWave radar U-V pixel coordinate system. We evaluate the system with experiments on 12 subjects in three scenarios, and the results across four metrics demonstrate the effectiveness of our hand localization system.

References in corpus (9)

• Knowledge Distillation: A Survey
• FitNets: Hints for Thin Deep Nets
• Human Action Recognition from Various Data Modalities: A Review
• MediaPipe Hands: On-device Real-time Hand Tracking
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Different Approaches for Human Activity Recognition: A Survey
• Product Inspection Methodology via Deep Learning: An Overview
• Beyond-Voice: Towards Continuous 3D Hand Pose Tracking on Commercial Home Assistant Devices
• Synthetic training data generation for deep learning based quality inspection