Latent Relation Language Models

Abstract

In this paper, we propose Latent Relation Language Models (LRLMs), a class of language models that parameterizes the joint distribution over the words in a document and the entities that occur therein via knowledge graph relations. This model has a number of attractive properties: it not only improves language modeling performance, but is also able to annotate the posterior probability of entity spans for a given text through relations. Experiments demonstrate empirical improvements over both a word-based baseline language model and a previous approach that incorporates knowledge graph information. Qualitative analysis further demonstrates the proposed model's ability to learn to predict appropriate relations in context.

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• A Neural Knowledge Language Model
• Learning to Compute Word Embeddings On the Fly
• Neural Machine Translation with External Phrase Memory
• Describing a Knowledge Base

Cited by in corpus (5)

• How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
• Knowledge Enhanced Pretrained Language Models: A Compreshensive Survey
• Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
• Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths
• Knowledge Graph Completion with Text-aided Regularization