Long Short-Term Attention
arXiv:1810.12752
Abstract
Attention is an important cognition process of humans, which helps humans concentrate on critical information during their perception and learning. However, although many machine learning models can remember information of data, they have no the attention mechanism. For example, the long short-term memory (LSTM) network is able to remember sequential information, but it cannot pay special attention to part of the sequences. In this paper, we present a novel model called long short-term attention (LSTA), which seamlessly integrates the attention mechanism into the inner cell of LSTM. More than processing long short term dependencies, LSTA can focus on important information of the sequences with the attention mechanism. Extensive experiments demonstrate that LSTA outperforms LSTM and related models on the sequence learning tasks.
References in corpus (16)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- LSTM: A Search Space Odyssey
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- A Critical Review of Recurrent Neural Networks for Sequence Learning
- A Structured Self-attentive Sentence Embedding
- A Deep Reinforced Model for Abstractive Summarization
- Exploring the Limits of Language Modeling
- End-To-End Memory Networks
- Learning Structured Sparsity in Deep Neural Networks
- Long Short-Term Memory-Networks for Machine Reading
- Structured Attention Networks
- A Decomposable Attention Model for Natural Language Inference
- Wider and Deeper, Cheaper and Faster: Tensorized LSTMs for Sequence Learning