Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
arXiv:2004.14224
Abstract
In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.
References in corpus (7)
- A Novel Embedding Model for Knowledge Base Completion Based on Convolutional Neural Network
- ERNIE: Enhanced Representation through Knowledge Integration
- RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
- Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
- KG-BERT: BERT for Knowledge Graph Completion
- Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
- Commonsense Knowledge Mining from Pretrained Models