Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition

Abstract

Integrating external language models (LMs) into end-to-end (E2E) models remains a challenging task for domain-adaptive speech recognition. Recently, internal language model estimation (ILME)-based LM fusion has shown significant word error rate (WER) reduction from Shallow Fusion by subtracting a weighted internal LM score from an interpolation of E2E model and external LM scores during beam search. However, on different test sets, the optimal LM interpolation weights vary over a wide range and have to be tuned extensively on well-matched validation sets. In this work, we perform LM fusion in the minimum WER (MWER) training of an E2E model to obviate the need for LM weights tuning during inference. Besides MWER training with Shallow Fusion (MWER-SF), we propose a novel MWER training with ILME (MWER-ILME) where the ILME-based fusion is conducted to generate N-best hypotheses and their posteriors. Additional gradient is induced when internal LM is engaged in MWER-ILME loss computation. During inference, LM weights pre-determined in MWER training enable robust LM integrations on test sets from different domains. Experimented with 30K-hour trained transformer transducers, MWER-ILME achieves on average 8.8% and 5.8% relative WER reductions from MWER and MWER-SF training, respectively, on 6 different test sets

arXiv comment5 pages, Interspeech 2021

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• On Using Monolingual Corpora in Neural Machine Translation
• Common Voice: A Massively-Multilingual Speech Corpus
• Speaker-Invariant Training via Adversarial Learning
• Conditional Teacher-Student Learning
• Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition
• Adversarial Teacher-Student Learning for Unsupervised Domain Adaptation
• Adaptation Algorithms for Neural Network-Based Speech Recognition: An Overview
• Unsupervised Adaptation with Domain Separation Networks for Robust Speech Recognition
• Invariant Representations for Noisy Speech Recognition
• Adversarial Speaker Adaptation
• Minimum Bayes Risk Training of RNN-Transducer for End-to-End Speech Recognition
• Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Dataset
• On Minimum Word Error Rate Training of the Hybrid Autoregressive Transducer