Attention-Based Models for Speech Recognition

Abstract

Recurrent sequence generators conditioned on input data through an attention mechanism have recently shown very good performance on a range of tasks in- cluding machine translation, handwriting synthesis and image caption gen- eration. We extend the attention-mechanism with features needed for speech recognition. We show that while an adaptation of the model used for machine translation in reaches a competitive 18.7% phoneme error rate (PER) on the TIMIT phoneme recognition task, it can only be applied to utterances which are roughly as long as the ones it was trained on. We offer a qualitative explanation of this failure and propose a novel and generic method of adding location-awareness to the attention mechanism to alleviate this issue. The new method yields a model that is robust to long inputs and achieves 18% PER in single utterances and 20% in 10-times longer (repeated) utterances. Finally, we propose a change to the at- tention mechanism that prevents it from concentrating too much on single frames, which further reduces PER to 17.6% level.

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• ADADELTA: An Adaptive Learning Rate Method
• Sequence Transduction with Recurrent Neural Networks
• Theano: new features and speed improvements
• Recurrent Models of Visual Attention
• On Using Monolingual Corpora in Neural Machine Translation
• End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results
• Blocks and Fuel: Frameworks for deep learning
• Neural Turing Machines

Cited by in corpus (2)

• Multichannel End-to-end Speech Recognition
• Attention and Localization based on a Deep Convolutional Recurrent Model for Weakly Supervised Audio Tagging