Long Short-Term Memory with Dynamic Skip Connections
arXiv:1811.03873
Abstract
In recent years, long short-term memory (LSTM) has been successfully used to model sequential data of variable length. However, LSTM can still experience difficulty in capturing long-term dependencies. In this work, we tried to alleviate this problem by introducing a dynamic skip connection, which can learn to directly connect two dependent words. Since there is no dependency information in the training data, we propose a novel reinforcement learning-based method to model the dependency relationship and connect dependent words. The proposed model computes the recurrent transition functions based on the skip connections, which provides a dynamic skipping advantage over RNNs that always tackle entire sentences sequentially. Our experimental results on three natural language processing tasks demonstrate that the proposed method can achieve better performance than existing methods. In the number prediction experiment, the proposed model outperformed LSTM with respect to accuracy by nearly 20%.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Recurrent Neural Network Regularization
- Character-Aware Neural Language Models
- Pointer Sentinel Mixture Models
- Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
- Hierarchical Multiscale Recurrent Neural Networks
- How to Construct Deep Recurrent Neural Networks
- Bridging the Gap Between Value and Policy Based Reinforcement Learning
- Modeling Coverage for Neural Machine Translation
- Dilated Recurrent Neural Networks
- Architectural Complexity Measures of Recurrent Neural Networks
- Neural Speed Reading via Skim-RNN
- Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks