ERNIE: Enhanced Language Representation with Informative Entities

Abstract

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The source code of this paper can be obtained from https://github.com/thunlp/ERNIE.

Accepted by ACL 2019

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• Gaussian Error Linear Units (GELUs)
• ERNIE: Enhanced Representation through Knowledge Integration
• Semi-supervised Sequence Learning
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Joint Representation Learning of Text and Knowledge for Knowledge Graph Completion
• Improving Question Answering by Commonsense-Based Pre-Training

Cited by in corpus (25)

• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Downstream Model Design of Pre-trained Language Model for Relation Extraction Task
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
• A BERT based Sentiment Analysis and Key Entity Detection Approach for Online Financial Texts
• E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
• Retrieval-Augmented Transformer-XL for Close-Domain Dialog Generation
• BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance
• Evaluation of Sentence Representations in Polish
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• Superbloom: Bloom filter meets Transformer
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrifice
• Billion-scale Pre-trained E-commerce Product Knowledge Graph Model
• Sequential Diagnosis Prediction with Transformer and Ontological Representation
• Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Learning Better Sentence Representation with Syntax Information
• Pentagon at MEDIQA 2019: Multi-task Learning for Filtering and Re-ranking Answers using Language Inference and Question Entailment
• A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generation
• Quantization Aware Training, ERNIE and Kurtosis Regularizer: a short empirical study