Enabling hand gesture customization on wrist-worn devices

Abstract

We present a framework for gesture customization requiring minimal examples from users, all without degrading the performance of existing gesture sets. To achieve this, we first deployed a large-scale study (N=500+) to collect data and train an accelerometer-gyroscope recognition model with a cross-user accuracy of 95.7% and a false-positive rate of 0.6 per hour when tested on everyday non-gesture data. Next, we design a few-shot learning framework which derives a lightweight model from our pre-trained model, enabling knowledge transfer without performance degradation. We validate our approach through a user study (N=20) examining on-device customization from 12 new gestures, resulting in an average accuracy of 55.3%, 83.1%, and 87.2% on using one, three, or five shots when adding a new gesture, while maintaining the same recognition accuracy and false-positive rate from the pre-existing gesture set. We further evaluate the usability of our real-time implementation with a user experience study (N=20). Our results highlight the effectiveness, learnability, and usability of our customization framework. Our approach paves the way for a future where users are no longer bound to pre-existing gestures, freeing them to creatively introduce new gestures tailored to their preferences and abilities.

Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

References in corpus (2)

• Federated Learning: Challenges, Methods, and Future Directions
• An Empirical Survey of Data Augmentation for Time Series Classification with Neural Networks

Cited by in corpus (10)

• Time2Stop: Adaptive and Explainable Human-AI Loop for Smartphone Overuse Intervention
• EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a Wristband
• LipLearner: Customizable Silent Speech Interactions on Mobile Devices
• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• picoRing: battery-free rings for subtle thumb-to-index input
• GestureGPT: Toward Zero-Shot Free-Form Hand Gesture Understanding with Large Language Model Agents
• DanmuA11y: Making Time-Synced On-Screen Video Comments (Danmu) Accessible to Blind and Low Vision Users via Multi-Viewer Audio Discussions
• Vision-Based Hand Gesture Customization from a Single Demonstration
• Single-tap Latency Reduction with Single- or Double- tap Prediction
• Experimental Shake Gesture Detection API for Apple Watch