RescoreBERT: Discriminative Speech Recognition Rescoring with BERT

Abstract

Second-pass rescoring is an important component in automatic speech recognition (ASR) systems that is used to improve the outputs from a first-pass decoder by implementing a lattice rescoring or $n$-best re-ranking. While pretraining with a masked language model (MLM) objective has received great success in various natural language understanding (NLU) tasks, it has not gained traction as a rescoring model for ASR. Specifically, training a bidirectional model like BERT on a discriminative objective such as minimum WER (MWER) has not been explored. Here we show how to train a BERT-based rescoring model with MWER loss, to incorporate the improvements of a discriminative loss into fine-tuning of deep bidirectional pretrained models for ASR. Specifically, we propose a fusion strategy that incorporates the MLM into the discriminative training process to effectively distill knowledge from a pretrained model. We further propose an alternative discriminative loss. This approach, which we call RescoreBERT, reduces WER by 6.6%/3.4% relative on the LibriSpeech clean/other test sets over a BERT baseline without discriminative objective. We also evaluate our method on an internal dataset from a conversational agent and find that it reduces both latency and WER (by 3 to 8% relative) over an LSTM rescoring model.

Accepted to ICASSP 2022

References in corpus (3)

• Distilling the Knowledge in a Neural Network
• On Using Monolingual Corpora in Neural Machine Translation
• Innovative Bert-based Reranking Language Models for Speech Recognition

Cited by in corpus (6)

• Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting
• Low-rank Adaptation of Large Language Model Rescoring for Parameter-Efficient Speech Recognition
• Exploring the Integration of Large Language Models into Automatic Speech Recognition Systems: An Empirical Study
• PROCTER: PROnunciation-aware ConTextual adaptER for personalized speech recognition in neural transducers
• Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunities
• On Comparison of Encoders for Attention based End to End Speech Recognition in Standalone and Rescoring Mode