Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition

Abstract

Adapting a trained Automatic Speech Recognition (ASR) model to new tasks results in catastrophic forgetting of old tasks, limiting the model's ability to learn continually and to be extended to new speakers, dialects, languages, etc. Focusing on End-to-End ASR, in this paper, we propose a simple yet effective method to overcome catastrophic forgetting: weight averaging. By simply taking the average of the previous and the adapted model, our method achieves high performance on both the old and new tasks. It can be further improved by introducing a knowledge distillation loss during the adaptation. We illustrate the effectiveness of our method on both monolingual and multilingual ASR. In both cases, our method strongly outperforms all baselines, even in its simplest form.

Accepted at ICASSP 2023. 5 pages

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Continual Lifelong Learning in Natural Language Processing: A Survey
• Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition
• Continual Learning for Monolingual End-to-End Automatic Speech Recognition
• Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging

Cited by in corpus (2)

• The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies
• Inverse-Hessian Regularization for Continual Learning in ASR