Personalizing ASR for Dysarthric and Accented Speech with Limited Data

Abstract

Automatic speech recognition (ASR) systems have dramatically improved over the last few years. ASR systems are most often trained from 'typical' speech, which means that underrepresented groups don't experience the same level of improvement. In this paper, we present and evaluate finetuning techniques to improve ASR for users with non-standard speech. We focus on two types of non-standard speech: speech from people with amyotrophic lateral sclerosis (ALS) and accented speech. We train personalized models that achieve 62% and 35% relative WER improvement on these two groups, bringing the absolute WER for ALS speakers, on a test set of message bank phrases, down to 10% for mild dysarthria and 20% for more serious dysarthria. We show that 71% of the improvement comes from only 5 minutes of training data. Finetuning a particular subset of layers (with many fewer parameters) often gives better results than finetuning the entire model. This is the first step towards building state of the art ASR models for dysarthric speech.

arXiv comment5 pages

References in corpus (5)

• How transferable are features in deep neural networks?
• Sequence Transduction with Recurrent Neural Networks
• Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• Articulatory Features for ASR of Pathological Speech

Cited by in corpus (19)

• Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition
• Universal Paralinguistic Speech Representations Using Self-Supervised Conformers
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognition
• Accented Speech Recognition: A Survey
• Variational Auto-Encoder Based Variability Encoding for Dysarthric Speech Recognition
• On-Device Personalization of Automatic Speech Recognition Models for Disordered Speech
• Exploring and Evaluating Personalized Models for Code Generation
• End-to-End Whisper to Natural Speech Conversion using Modified Transformer Network
• Improved End-to-End Dysarthric Speech Recognition via Meta-learning Based Model Re-initialization
• A Scalable Model Specialization Framework for Training and Inference using Submodels and its Application to Speech Model Personalization
• Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition
• DITTO: Data-efficient and Fair Targeted Subset Selection for ASR Accent Adaptation
• Significance of Data Augmentation for Improving Cleft Lip and Palate Speech Recognition
• ASR Personalization in Pathological Voice: Accuracy, Efficiency, and Duration Effects
• End-to-End Simultaneous Dysarthric Speech Reconstruction with Frame-Level Adaptor and Multiple Wait-k Knowledge Distillation
• Optimal Transport-based Adaptation in Dysarthric Speech Tasks
• Learning Explicit Prosody Models and Deep Speaker Embeddings for Atypical Voice Conversion
• Gender domain adaptation for automatic speech recognition task