Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR
arXiv:2607.11163
The paper introduces Unified Gradient Projection (UGP), a method that uses language‑balanced replay gradients to constrain parameter updates, reducing dominant‑language bias and catastrophic forgetting when fine‑tuning large multilingual ASR models on low‑resource languages.
Abstract
Large-scale pretrained ASR models such as Whisper exhibit strong multilingual capabilities. However, fine-tuning on low-resource languages often causes catastrophic forgetting. Although continual learning mitigates this issue, existing methods struggle to regulate cross-task interference in multilingual settings, where dominant languages bias optimization. We propose Unified Gradient Projection (UGP), which constrains parameter updates using reference gradients from language-balanced replay in a unified projection space. By equalizing per-language contributions in the projection, UGP reduces dominant-language bias and improves cross-lingual stability. We further show that combining gradient-level projection with data-level replay yields complementary gains in stability and plasticity. Across diverse low-resource language groups and model scales, UGP enables effective adaptation while substantially mitigating forgetting. On Whisper-large-v3, it achieves near-zero average forgetting.
Accepted by Interspeech 2026