A Neural Knowledge Language Model
arXiv:1608.00318
Abstract
Current language models have a significant limitation in the ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge from statistical co-occurrences although most of the knowledge words are rarely observed. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by the knowledge graph with the RNN language model. By predicting whether the word to generate has an underlying fact or not, the model can generate such knowledge-related words by copying from the description of the predicted fact. In experiments, we show that the NKLM significantly improves the performance while generating a much smaller number of unknown words.
References in corpus (15)
- Recurrent Neural Network Regularization
- A Review of Relational Machine Learning for Knowledge Graphs
- A Neural Conversational Model
- Theano: new features and speed improvements
- Exploring the Limits of Language Modeling
- End-To-End Memory Networks
- Large-scale Simple Question Answering with Memory Networks
- The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations
- On Using Monolingual Corpora in Neural Machine Translation
- Pointer Sentinel Mixture Models
- Incorporating Copying Mechanism in Sequence-to-Sequence Learning
- Dialog-based Language Learning
- Neural Generative Question Answering
- Leveraging Lexical Resources for Learning Entity Embeddings in Multi-Relational Data
- Row-less Universal Schema
Cited by in corpus (41)
- A Survey on Knowledge Graphs: Representation, Acquisition and Applications
- Neural Natural Language Inference Models Enhanced with External Knowledge
- Representation Learning for Natural Language Processing
- Topic-Guided Variational Autoencoders for Text Generation
- Knowledge-Aware Language Model Pretraining
- A Survey on Neural Network Language Models
- Linguistic Knowledge as Memory for Recurrent Neural Networks
- Leveraging Knowledge Bases in LSTMs for Improving Machine Reading
- Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
- Knowledge Representation Learning: A Quantitative Review
- Affect-LM: A Neural Language Model for Customizable Affective Text Generation
- Self-Knowledge Distillation in Natural Language Processing
- Question Answering from Unstructured Text by Retrieval and Comprehension
- Learning in Text Streams: Discovery and Disambiguation of Entity and Relation Instances
- Improving Question Answering by Commonsense-Based Pre-Training
- Knowledge Enhanced Contextual Word Representations
- Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
- Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs
- Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths
- Describing Natural Images Containing Novel Objects with Knowledge Guided Assitance
- Machine Learning with World Knowledge: The Position and Survey
- Knowledge-Augmented Language Model and its Application to Unsupervised Named-Entity Recognition
- Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
- WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web
- Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
- MMKG: Multi-Modal Knowledge Graphs
- Latent Relation Language Models
- KGPT: Knowledge-Grounded Pre-Training for Data-to-Text Generation
- Solving Cold-Start Problem in Large-scale Recommendation Engines: A Deep Learning Approach
- CoTK: An Open-Source Toolkit for Fast Development and Fair Evaluation of Text Generation
- Be Concise and Precise: Synthesizing Open-Domain Entity Descriptions from Facts
- TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts
- Knowledge Graph Embedding with Multiple Relation Projections
- Contextual Text Embeddings for Twi
- Aff2Vec: Affect--Enriched Distributional Word Representations
- Cooperative Reasoning on Knowledge Graph and Corpus: A Multi-agentReinforcement Learning Approach
- PoMo: Generating Entity-Specific Post-Modifiers in Context
- A Neural Turing~Machine for Conditional Transition Graph Modeling
- Everything Happens for a Reason: Discovering the Purpose of Actions in Procedural Text
- SAM: Semantic Attribute Modulation for Language Modeling and Style Variation
- Barack's Wife Hillary: Using Knowledge-Graphs for Fact-Aware Language Modeling