RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions

Abstract

In recent years, all-neural end-to-end approaches have obtained state-of-the-art results on several challenging automatic speech recognition (ASR) tasks. However, most existing works focus on building ASR models where train and test data are drawn from the same domain. This results in poor generalization characteristics on mismatched-domains: e.g., end-to-end models trained on short segments perform poorly when evaluated on longer utterances. In this work, we analyze the generalization properties of streaming and non-streaming recurrent neural network transducer (RNN-T) based end-to-end models in order to identify model components that negatively affect generalization performance. We propose two solutions: combining multiple regularization techniques during training, and using dynamic overlapping inference. On a long-form YouTube test set, when the nonstreaming RNN-T model is trained with shorter segments of data, the proposed combination improves word error rate (WER) from 22.3% to 14.8%; when the streaming RNN-T model trained on short Search queries, the proposed techniques improve WER on the YouTube set from 67.0% to 25.3%. Finally, when trained on Librispeech, we find that dynamic overlapping inference improves WER on YouTube from 99.8% to 33.0%.

arXiv commentSLT camera-ready version

References in corpus (7)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
• On the Compression of Recurrent Neural Networks with an Application to LVCSR acoustic modeling for Embedded Speech Recognition
• A comparison of end-to-end models for long-form speech recognition

Cited by in corpus (6)

• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• A New Training Pipeline for an Improved Neural Transducer
• Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models
• Hypothesis Stitcher for End-to-End Speaker-attributed ASR on Long-form Multi-talker Recordings
• Improved Neural Language Model Fusion for Streaming Recurrent Neural Network Transducer
• Improving Streaming Automatic Speech Recognition With Non-Streaming Model Distillation On Unsupervised Data