Inverse-Hessian Regularization for Continual Learning in ASR

Abstract

Catastrophic forgetting remains a major challenge for continual learning (CL) in automatic speech recognition (ASR), where models must adapt to new domains without losing performance on previously learned conditions. Several CL methods have been proposed for ASR, and, recently, weight averaging - where models are averaged in a merging step after fine-tuning - has proven effective as a simple memory-free strategy. However, it is heuristic in nature and ignores the underlying loss landscapes of the tasks, hindering adaptability. In this work, we propose Inverse Hessian Regularization (IHR), a memory-free approach for CL in ASR that incorporates curvature information into the merging step. After fine-tuning on a new task, the adaptation is adjusted through a Kronecker-factored inverse Hessian approximation of the previous task, ensuring that the model moves primarily in directions less harmful to past performance, while keeping the method lightweight. We evaluate IHR on two CL benchmarks and show that it significantly outperforms state-of-the-art baselines, reducing forgetting while improving adaptability. Ablation studies and analyses further confirm its effectiveness.

Accepted for presentation at ICASSP 2026

References in corpus (10)

• Overcoming catastrophic forgetting in neural networks
• A continual learning survey: Defying forgetting in classification tasks
• Continual Lifelong Learning in Natural Language Processing: A Survey
• Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognition
• Libri-Adapt: A New Speech Dataset for Unsupervised Domain Adaptation
• Continual Learning for Monolingual End-to-End Automatic Speech Recognition
• Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognition
• Continual Learning for On-Device Speech Recognition using Disentangled Conformers
• Unsupervised Online Continual Learning for Automatic Speech Recognition
• Continual Learning With Quasi-Newton Methods