Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition

Abstract

We propose a neural language modeling system based on low-rank adaptation (LoRA) for speech recognition output rescoring. Although pretrained language models (LMs) like BERT have shown superior performance in second-pass rescoring, the high computational cost of scaling up the pretraining stage and adapting the pretrained models to specific domains limit their practical use in rescoring. Here we present a method based on low-rank decomposition to train a rescoring BERT model and adapt it to new domains using only a fraction (0.08%) of the pretrained parameters. These inserted matrices are optimized through a discriminative training objective along with a correlation-based regularization loss. The proposed low-rank adaptation Rescore-BERT (LoRB) architecture is evaluated on LibriSpeech and internal datasets with decreased training times by factors between 5.4 and 3.6.

arXiv commentAccepted to IEEE ASRU 2023. Internal Review Approved. Revised 2nd version with Andreas and Huck. The first version is in Sep 29th. 8 pages

References in corpus (11)

• SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
• RescoreBERT: Discriminative Speech Recognition Rescoring with BERT
• One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuning
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition
• SpeechPrompt v2: Prompt Tuning for Speech Classification Tasks
• PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers
• Voice2Series: Reprogramming Acoustic Models for Time Series Classification
• Parameter-Efficient Learning for Text-to-Speech Accent Adaptation
• How to Estimate Model Transferability of Pre-Trained Speech Models?
• Neural Model Reprogramming with Similarity Based Mapping for Low-Resource Spoken Command Recognition
• Differentially Private Adapters for Parameter Efficient Acoustic Modeling

Cited by in corpus (3)

• Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition
• Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning
• Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches