Continual Learning for Monolingual End-to-End Automatic Speech Recognition

Abstract

Adapting Automatic Speech Recognition (ASR) models to new domains results in a deterioration of performance on the original domain(s), a phenomenon called Catastrophic Forgetting (CF). Even monolingual ASR models cannot be extended to new accents, dialects, topics, etc. without suffering from CF, making them unable to be continually enhanced without storing all past data. Fortunately, Continual Learning (CL) methods, which aim to enable continual adaptation while overcoming CF, can be used. In this paper, we implement an extensive number of CL methods for End-to-End ASR and test and compare their ability to extend a monolingual Hybrid CTC-Transformer model across four new tasks. We find that the best performing CL method closes the gap between the fine-tuned model (lower bound) and the model trained jointly on all tasks (upper bound) by more than 40%, while requiring access to only 0.6% of the original data.

arXiv commentPublished at EUSIPCO 2022. 5 pages, 1 figure

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• A continual learning survey: Defying forgetting in classification tasks
• Efficient Lifelong Learning with A-GEM
• Gradient Episodic Memory for Continual Learning
• Experience Replay for Continual Learning
• On Tiny Episodic Memories in Continual Learning
• Batch-level Experience Replay with Review for Continual Learning

Cited by in corpus (5)

• Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition
• Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition
• Continual Learning for On-Device Speech Recognition using Disentangled Conformers
• Replay to Remember: Continual Layer-Specific Fine-tuning for German Speech Recognition
• Inverse-Hessian Regularization for Continual Learning in ASR