Linguistic Knowledge as Memory for Recurrent Neural Networks

Abstract

Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model which memories it should utilize. Specifically, external knowledge is used to augment a sequence with typed edges between arbitrarily distant elements, and the resulting graph is decomposed into directed acyclic subgraphs. We introduce a model that encodes such graphs as explicit memory in recurrent neural networks, and use it to model coreference relations in text. We apply our model to several text comprehension tasks and achieve new state-of-the-art results on all considered benchmarks, including CNN, bAbi, and LAMBADA. On the bAbi QA tasks, our model solves 15 out of the 20 tasks with only 1000 training examples per task. Analysis of the learned representations further demonstrates the ability of our model to encode fine-grained entity information across a document.

References in corpus (7)

• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• A Neural Knowledge Language Model
• Query-Reduction Networks for Question Answering
• Who did What: A Large-Scale Person-Centered Cloze Dataset
• Reference-Aware Language Models
• Broad Context Language Modeling as Reading Comprehension

Cited by in corpus (14)

• Dynamic Integration of Background Knowledge in Neural NLU Systems
• Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks
• Graph Convolutional Networks for Named Entity Recognition
• Establishing Baselines for Text Classification in Low-Resource Languages
• Graph Neural Networks with Generated Parameters for Relation Extraction
• Discourse-Aware Semantic Self-Attention for Narrative Reading Comprehension
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• A Comparative Study of Rule Extraction for Recurrent Neural Networks
• Evidence Sentence Extraction for Machine Reading Comprehension
• An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks
• On-The-Fly Information Retrieval Augmentation for Language Models
• Knowledge Efficient Deep Learning for Natural Language Processing
• On Computability, Learnability and Extractability of Finite State Machines from Recurrent Neural Networks
• Multi-hop Reading Comprehension via Deep Reinforcement Learning based Document Traversal