Understanding Personalization Gains in Whisper ASR

Abstract

Whisper-style ASR models provide strong zero-shot transcription, but their residual errors are not evenly distributed across datasets, recording conditions, and metadata-based groups. We study when lightweight low-rank adaptation (LoRA) improves Whisper and whether observed gains require group-specific adapters. Across Common Voice, LibriSpeech, and VoxPopuli, we compare zero-shot Whisper, pooled LoRA, and group-specific LoRA under matched within-group splits. We find that adaptation is not uniformly beneficial: gains are concentrated in difficult Common Voice groups, are consistent on VoxPopuli, and largely disappear on low-error LibriSpeech. Crucially, pooled LoRA often matches, and sometimes clearly outperforms, group-specific LoRA. Gains over zero-shot should therefore be tested against matched pooled adaptation before being interpreted as evidence of personalization.

arXiv comment14 pages. Presented at SPECOM 2026

References in corpus (9)

• LoRA: Low-Rank Adaptation of Large Language Models
• Robust Speech Recognition via Large-Scale Weak Supervision
• Learning Hidden Unit Contributions for Unsupervised Acoustic Model Adaptation
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Speaker Adaptation for Attention-Based End-to-End Speech Recognition
• Variational Low-Rank Adaptation for Personalized Impaired Speech Recognition
• PI-Whisper: Designing an Adaptive and Incremental Automatic Speech Recognition System for Edge Devices
• Speaker Adaptation for Quantised End-to-End ASR Models
• Two-Stage Adaptation for Non-Normative Speech Recognition: Revisiting Speaker-Independent Initialization for Personalization