EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband

Abstract

Our hands serve as a fundamental means of interaction with the world around us. Therefore, understanding hand poses and interaction context is critical for human-computer interaction. We present EchoWrist, a low-power wristband that continuously estimates 3D hand pose and recognizes hand-object interactions using active acoustic sensing. EchoWrist is equipped with two speakers emitting inaudible sound waves toward the hand. These sound waves interact with the hand and its surroundings through reflections and diffractions, carrying rich information about the hand's shape and the objects it interacts with. The information captured by the two microphones goes through a deep learning inference system that recovers hand poses and identifies various everyday hand activities. Results from the two 12-participant user studies show that EchoWrist is effective and efficient at tracking 3D hand poses and recognizing hand-object interactions. Operating at 57.9mW, EchoWrist is able to continuously reconstruct 20 3D hand joints with MJEDE of 4.81mm and recognize 12 naturalistic hand-object interactions with 97.6% accuracy.

References in corpus (4)

• Enabling hand gesture customization on wrist-worn devices
• EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses
• GazeTrak: Exploring Acoustic-based Eye Tracking on a Glass Frame
• Continuous Gesture Recognition from sEMG Sensor Data with Recurrent Neural Networks and Adversarial Domain Adaptation

Cited by in corpus (9)

• EyeEcho: Continuous and Low-power Facial Expression Tracking on Glasses
• GazeTrak: Exploring Acoustic-based Eye Tracking on a Glass Frame
• SpellRing: Recognizing Continuous Fingerspelling in American Sign Language using a Ring
• MunchSonic: Tracking Fine-grained Dietary Actions through Active Acoustic Sensing on Eyeglasses
• Wrist2Finger: Sensing Fingertip Force for Force-Aware Hand Interaction with a Ring-Watch Wearable
• EchoForce: Continuous Grip Force Estimation from Skin Deformation Using Active Acoustic Sensing on a Wristband
• VibWalk: Mapping Lower-limb Haptic Experiences of Everyday Walking
• Grab-n-Go: On-the-Go Microgesture Recognition with Objects in Hand
• SensPS: Sensing Personal Space Comfortable Distance between Human-Human Using Multimodal Sensors