Recurrent Memory Networks for Language Modeling
arXiv:1601.01272
Abstract
Recurrent Neural Networks (RNN) have obtained excellent result in many natural language processing (NLP) tasks. However, understanding and interpreting the source of this success remains a challenge. In this paper, we propose Recurrent Memory Network (RMN), a novel RNN architecture, that not only amplifies the power of RNN but also facilitates our understanding of its internal functioning and allows us to discover underlying patterns in data. We demonstrate the power of RMN on language modeling and sentence completion tasks. On language modeling, RMN outperforms Long Short-Term Memory (LSTM) network on three large German, Italian, and English dataset. Additionally we perform in-depth analysis of various linguistic dimensions that RMN captures. On Sentence Completion Challenge, for which it is essential to capture sentence coherence, our RMN obtains 69.2% accuracy, surpassing the previous state-of-the-art by a large margin.
arXiv comment8 pages, 6 figures. Accepted at NAACL 2016
References in corpus (8)
- Efficient Estimation of Word Representations in Vector Space
- Sequence to Sequence Learning with Neural Networks
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- LSTM: A Search Space Odyssey
- On the difficulty of training Recurrent Neural Networks
- Visualizing and Understanding Recurrent Networks
- Grid Long Short-Term Memory
- Strategies for Training Large Vocabulary Neural Language Models
Cited by in corpus (15)
- Recent Advances in Recurrent Neural Networks
- Long Short-Term Memory-Networks for Machine Reading
- R-Transformer: Recurrent Neural Network Enhanced Transformer
- Learning Python Code Suggestion with a Sparse Pointer Network
- Attentive Memory Networks: Efficient Machine Reading for Conversational Search
- Neural Language Generation: Formulation, Methods, and Evaluation
- Self-Attentive Residual Decoder for Neural Machine Translation
- Learning to Remember Translation History with a Continuous Cache
- The Importance of Being Recurrent for Modeling Hierarchical Structure
- Unsupervised Neural Hidden Markov Models
- Reflective Decoding Network for Image Captioning
- Joint Embedding Learning of Educational Knowledge Graphs
- Attention-based Memory Selection Recurrent Network for Language Modeling
- Meta-Learning a Dynamical Language Model
- Improving Neural Language Models by Segmenting, Attending, and Predicting the Future