Listen with Intent: Improving Speech Recognition with Audio-to-Intent Front-End

Abstract

Comprehending the overall intent of an utterance helps a listener recognize the individual words spoken. Inspired by this fact, we perform a novel study of the impact of explicitly incorporating intent representations as additional information to improve a recurrent neural network-transducer (RNN-T) based automatic speech recognition (ASR) system. An audio-to-intent (A2I) model encodes the intent of the utterance in the form of embeddings or posteriors, and these are used as auxiliary inputs for RNN-T training and inference. Experimenting with a 50k-hour far-field English speech corpus, this study shows that when running the system in non-streaming mode, where intent representation is extracted from the entire utterance and then used to bias streaming RNN-T search from the start, it provides a 5.56% relative word error rate reduction (WERR). On the other hand, a streaming system using per-frame intent posteriors as extra inputs for the RNN-T ASR system yields a 3.33% relative WERR. A further detailed analysis of the streaming system indicates that our proposed method brings especially good gain on media-playing related intents (e.g. 9.12% relative WERR on PlayMusicIntent).

arXiv commentTo appear in Interspeech 2021

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Attention-Based Models for Speech Recognition
• Sequence Transduction with Recurrent Neural Networks
• Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfaces
• Attention-based Transducer for Online Speech Recognition
• Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capability