LipLearner: Customizable Silent Speech Interactions on Mobile Devices
arXiv:2302.05907 · doi:10.1145/3544548.3581465
Abstract
Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.
arXiv commentConditionally accepted to the ACM CHI Conference on Human Factors in Computing Systems 2023 (CHI '23)
References in corpus (7)
- Representation Learning with Contrastive Predictive Coding
- Learning Transferable Visual Models From Natural Language Supervision
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks
- Enabling hand gesture customization on wrist-worn devices
- Learn an Effective Lip Reading Model without Pains
- ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users