Gated Recurrent Context: Softmax-free Attention for Online Encoder-Decoder Speech Recognition

Abstract

Recently, attention-based encoder-decoder (AED) models have shown state-of-the-art performance in automatic speech recognition (ASR). As the original AED models with global attentions are not capable of online inference, various online attention schemes have been developed to reduce ASR latency for better user experience. However, a common limitation of the conventional softmax-based online attention approaches is that they introduce an additional hyperparameter related to the length of the attention window, requiring multiple trials of model training for tuning the hyperparameter. In order to deal with this problem, we propose a novel softmax-free attention method and its modified formulation for online attention, which does not need any additional hyperparameter at the training phase. Through a number of ASR experiments, we demonstrate the tradeoff between the latency and performance of the proposed online attention technique can be controlled by merely adjusting a threshold at the test phase. Furthermore, the proposed methods showed competitive performance to the conventional global and online attentions in terms of word-error-rates (WERs).

References in corpus (12)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Sequence Transduction with Recurrent Neural Networks
• Effective Approaches to Attention-based Neural Machine Translation
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Improved training of end-to-end attention models for speech recognition
• A Comprehensive Study of Deep Bidirectional LSTM RNNs for Acoustic Modeling in Speech Recognition
• RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
• Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
• A Neural Transducer
• Towards Online End-to-end Transformer Automatic Speech Recognition
• CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognition
• Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models