A Neural Knowledge Language Model
arXiv:1608.00318
Abstract
Current language models have a significant limitation in the ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge from statistical co-occurrences although most of the knowledge words are rarely observed. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by the knowledge graph with the RNN language model. By predicting whether the word to generate has an underlying fact or not, the model can generate such knowledge-related words by copying from the description of the predicted fact. In experiments, we show that the NKLM significantly improves the performance while generating a much smaller number of unknown words.
References in corpus (5)
Cited by in corpus (17)
- Topic-Guided Variational Autoencoders for Text Generation
- A Survey on Neural Network Language Models
- Linguistic Knowledge as Memory for Recurrent Neural Networks
- Leveraging Knowledge Bases in LSTMs for Improving Machine Reading
- Knowledge Representation Learning: A Quantitative Review
- Question Answering from Unstructured Text by Retrieval and Comprehension
- Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
- Describing Natural Images Containing Novel Objects with Knowledge Guided Assitance
- MMKG: Multi-Modal Knowledge Graphs
- Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach
- Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
- TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts
- Knowledge Graph Embedding with Multiple Relation Projections
- PoMo: Generating Entity-Specific Post-Modifiers in Context
- A Neural Turing~Machine for Conditional Transition Graph Modeling
- SAM: Semantic Attribute Modulation for Language Modeling and Style Variation
- Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling