A study of latent monotonic attention variants
arXiv:2103.16710
Abstract
End-to-end models reach state-of-the-art performance for speech recognition, but global soft attention is not monotonic, which might lead to convergence problems, to instability, to bad generalisation, cannot be used for online streaming, and is also inefficient in calculation. Monotonicity can potentially fix all of this. There are several ad-hoc solutions or heuristics to introduce monotonicity, but a principled introduction is rarely found in literature so far. In this paper, we present a mathematically clean solution to introduce monotonicity, by introducing a new latent variable which represents the audio position or segment boundaries. We compare several monotonic latent models to our global soft attention baseline such as a hard attention model, a local windowed soft attention model, and a segmental soft attention model. We can show that our monotonic models perform as good as the global soft attention model. We perform our experiments on Switchboard 300h. We carefully outline the details of our training and release our code and configs.
References in corpus (24)
- Sequence to Sequence Learning with Neural Networks
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
- Sequence Transduction with Recurrent Neural Networks
- Recurrent Models of Visual Attention
- Effective Approaches to Attention-based Neural Machine Translation
- Multiple Object Recognition with Visual Attention
- Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention
- Improved training of end-to-end attention models for speech recognition
- Online and Linear-Time Attention by Enforcing Monotonic Alignments
- Modeling Coverage for Neural Machine Translation
- Forward Attention in Sequence-to-sequence Acoustic Modelling for Speech Synthesis
- Latent Alignment and Variational Attention
- RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition
- Exploring Neural Transducers for End-to-End Speech Recognition
- Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
- Segmental Recurrent Neural Networks for End-to-end Speech Recognition
- A New Training Pipeline for an Improved Neural Transducer
- Monotonic Chunkwise Attention
- The Neural Noisy Channel
- Variational Attention for Sequence-to-Sequence Models
- Online Segment to Segment Neural Transduction
- An Online Attention-based Model for Speech Recognition
- Hard Non-Monotonic Attention for Character-Level Transduction