A Neural Knowledge Language Model

Abstract

Current language models have a significant limitation in the ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge from statistical co-occurrences although most of the knowledge words are rarely observed. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by the knowledge graph with the RNN language model. By predicting whether the word to generate has an underlying fact or not, the model can generate such knowledge-related words by copying from the description of the predicted fact. In experiments, we show that the NKLM significantly improves the performance while generating a much smaller number of unknown words.

References in corpus (5)

• Recurrent Neural Network Regularization
• Theano: new features and speed improvements
• Large-scale Simple Question Answering with Memory Networks
• On Using Monolingual Corpora in Neural Machine Translation
• Pointer Sentinel Mixture Models

Cited by in corpus (26)

• Representation Learning for Natural Language Processing
• Topic-Guided Variational Autoencoders for Text Generation
• A Survey on Neural Network Language Models
• Linguistic Knowledge as Memory for Recurrent Neural Networks
• Leveraging Knowledge Bases in LSTMs for Improving Machine Reading
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• Knowledge Representation Learning: A Quantitative Review
• Question Answering from Unstructured Text by Retrieval and Comprehension
• Knowledge Enhanced Contextual Word Representations
• Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths
• Describing Natural Images Containing Novel Objects with Knowledge Guided Assitance
• Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
• WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web
• Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
• MMKG: Multi-Modal Knowledge Graphs
• CoTK: An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
• Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach
• TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts
• Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
• Knowledge Graph Embedding with Multiple Relation Projections
• Contextual Text Embeddings for Twi
• PoMo: Generating Entity-Specific Post-Modifiers in Context
• Cooperative Reasoning on Knowledge Graph and Corpus: A Multi-agentReinforcement Learning Approach
• A Neural Turing~Machine for Conditional Transition Graph Modeling
• Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling
• SAM: Semantic Attribute Modulation for Language Modeling and Style Variation