Multi-scale Alignment and Contextual History for Attention Mechanism in Sequence-to-sequence Model
arXiv:1807.08280
Abstract
A sequence-to-sequence model is a neural network module for mapping two sequences of different lengths. The sequence-to-sequence model has three core modules: encoder, decoder, and attention. Attention is the bridge that connects the encoder and decoder modules and improves model performance in many tasks. In this paper, we propose two ideas to improve sequence-to-sequence model performance by enhancing the attention module. First, we maintain the history of the location and the expected context from several previous time-steps. Second, we apply multiscale convolution from several previous attention vectors to the current decoder state. We utilized our proposed framework for sequence-to-sequence speech recognition and text-to-speech systems. The results reveal that our proposed extension could improve performance significantly compared to a standard attention baseline.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Effective Approaches to Attention-based Neural Machine Translation
- Online and Linear-Time Attention by Enforcing Monotonic Alignments
- First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs
- Tacotron: Towards End-to-End Speech Synthesis
- Local Monotonic Attention Mechanism for End-to-End Speech and Language Processing
- Listening while Speaking: Speech Chain by Deep Learning
- Machine Speech Chain with One-shot Speaker Adaptation
- Attention-based Wav2Text with Feature Transfer Learning