ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users

Abstract

Recent advances have enabled automatic sound recognition systems for deaf and hard of hearing (DHH) users on mobile devices. However, these tools use pre-trained, generic sound recognition models, which do not meet the diverse needs of DHH users. We introduce ProtoSound, an interactive system for customizing sound recognition models by recording a few examples, thereby enabling personalized and fine-grained categories. ProtoSound is motivated by prior work examining sound awareness needs of DHH people and by a survey we conducted with 472 DHH participants. To evaluate ProtoSound, we characterized performance on two real-world sound datasets, showing significant improvement over state-of-the-art (e.g., +9.7% accuracy on the first dataset). We then deployed ProtoSound's end-user training and real-time recognition through a mobile application and recruited 19 hearing participants who listened to the real-world sounds and rated the accuracy across 56 locations (e.g., homes, restaurants, parks). Results show that ProtoSound personalized the model on-device in real-time and accurately learned sounds across diverse acoustic contexts. We close by discussing open challenges in personalizable sound recognition, including the need for better recording interfaces and algorithmic improvements.

arXiv commentPublished at the ACM CHI Conference on Human Factors in Computing Systems (CHI) 2022

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• Federated Machine Learning: Concept and Applications
• Applied Federated Learning: Improving Google Keyboard Query Suggestions
• Meta-learning for Few-shot Natural Language Processing: A Survey
• Unsupervised adversarial domain adaptation for acoustic scene classification

Cited by in corpus (8)

• LipLearner: Customizable Silent Speech Interactions on Mobile Devices
• SoundShift: Exploring Sound Manipulations for Accessible Mixed-Reality Awareness
• Exploring AI Problem Formulation with Children via Teachable Machines
• Super Kawaii Vocalics: Amplifying the "Cute" Factor in Computer Voice
• Meta-Learning in Audio and Speech Processing: An End to End Comprehensive Review
• HiSSNet: Sound Event Detection and Speaker Identification via Hierarchical Prototypical Networks for Low-Resource Headphones
• CapTune: Adapting Non-Speech Captions With Anchored Generative Models
• SonoCraftAR: Towards Supporting Personalized Authoring of Sound-Reactive AR Interfaces by Deaf and Hard of Hearing Users