ERNIE: Enhanced Representation through Knowledge Integration

Abstract

We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.

8 pages

References in corpus (6)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Cross-lingual Language Model Pretraining
• Universal Sentence Encoder
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Learning Semantic Textual Similarity from Conversations

Cited by in corpus (154)

• A Survey on Knowledge Graphs: Representation, Acquisition and Applications
• Pre-trained Models for Natural Language Processing: A Survey
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Deep Learning Based Text Classification: A Comprehensive Review
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
• Information Retrieval: Recent Advances and Beyond
• ERNIE: Enhanced Language Representation with Informative Entities
• Pretrained Domain-Specific Language Model for General Information Retrieval Tasks in the AEC Domain
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• A Survey on Text Classification: From Shallow to Deep Learning
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• NEZHA: Neural Contextualized Representation for Chinese Language Understanding
• K-BERT: Enabling Language Representation with Knowledge Graph
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• A Comprehensive Survey of Text Classification Techniques and Their Research Applications: Observational and Experimental Insights
• Align, Mask and Select: A Simple Method for Incorporating Commonsense Knowledge into Language Representation Models
• KERMIT: Generative Insertion-Based Modeling for Sequences
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• FastBERT: a Self-distilling BERT with Adaptive Inference Time
• Conceptualized Representation Learning for Chinese Biomedical Text Mining
• Label Information Enhanced Fraud Detection against Low Homophily in Graphs
• TNT-KID: Transformer-based Neural Tagger for Keyword Identification
• Modifying Memories in Transformer Models
• Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain
• Integrating Graph Contextualized Knowledge into Pre-trained Language Models
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Pre-train a Discriminative Text Encoder for Dense Retrieval via Contrastive Span Prediction
• The current status of large language models in summarizing radiology report impressions
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing
• ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• ERNIE-GEN: An Enhanced Multi-Flow Pre-training and Fine-tuning Framework for Natural Language Generation
• SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• DuTongChuan: Context-aware Translation Model for Simultaneous Interpreting
• Interpreting Deep Learning Models in Natural Language Processing: A Review
• Training Data Leakage Analysis in Language Models
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
• TranS^3: A Transformer-based Framework for Unifying Code Summarization and Code Search
• A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data
• A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation
• A Survey on Self-supervised Pre-training for Sequential Transfer Learning in Neural Networks
• DOBF: A Deobfuscation Pre-Training Objective for Programming Languages
• Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• To Tune or Not To Tune? How About the Best of Both Worlds?
• A Survey of Text Representation Methods and Their Genealogy
• Pre-Trained Models: Past, Present and Future
• Enriching BERT with Knowledge Graph Embeddings for Document Classification
• PMI-Masking: Principled masking of correlated spans
• Curriculum learning for language modeling
• When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset
• Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types
• ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations
• The Life Cycle of Knowledge in Big Language Models: A Survey
• GEDIT: Geographic-Enhanced and Dependency-Guided Tagging for Joint POI and Accessibility Extraction at Baidu Maps
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning
• InforMask: Unsupervised Informative Masking for Language Model Pretraining
• Synthetic QA Corpora Generation with Roundtrip Consistency
• Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
• Overview of the CCKS 2019 Knowledge Graph Evaluation Track: Entity, Relation, Event and QA
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• Training Multilingual Pre-trained Language Model with Byte-level Subwords
• Censorship of Online Encyclopedias: Implications for NLP Models
• KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation
• CBAG: Conditional Biomedical Abstract Generation
• Fine-tuning ERNIE for chest abnormal imaging signs extraction
• Lexicon Enhanced Chinese Sequence Labeling Using BERT Adapter
• Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension
• A Roadmap to Domain Knowledge Integration in Machine Learning
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• ERNIE at SemEval-2020 Task 10: Learning Word Emphasis Selection by Pre-trained Language Model
• ChineseBERT: Chinese Pretraining Enhanced by Glyph and Pinyin Information
• Does Knowledge Help General NLU? An Empirical Study
• Generating Representative Headlines for News Stories
• SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge
• LadRa-Net: Locally-Aware Dynamic Re-read Attention Net for Sentence Semantic Matching
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation
• Robust Optimization for Multilingual Translation with Imbalanced Data
• LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization
• Symmetric Regularization based BERT for Pair-wise Semantic Reasoning
• A Survey On Neural Word Embeddings
• DLGNet: A Transformer-based Model for Dialogue Response Generation
• Topology of Word Embeddings: Singularities Reflect Polysemy
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• Galileo at SemEval-2020 Task 12: Multi-lingual Learning for Offensive Language Identification using Pre-trained Language Models
• Which *BERT? A Survey Organizing Contextualized Encoders
• TSI: an Ad Text Strength Indicator using Text-to-CTR and Semantic-Ad-Similarity
• Application of Pre-training Models in Named Entity Recognition
• Learning Conceptual-Contextual Embeddings for Medical Text
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration
• Pre-trained Language Model for Web-scale Retrieval in Baidu Search
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• Neural Collaborative Graph Machines for Table Structure Recognition
• Machine Learning Meets Natural Language Processing -- The story so far
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• Heterogeneous Network Embedding for Deep Semantic Relevance Match in E-commerce Search
• ReadTwice: Reading Very Large Documents with Memories
• GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
• Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation
• OFAR: A Multimodal Evidence Retrieval Framework for Illegal Live-streaming Identification
• KFCNet: Knowledge Filtering and Contrastive Learning Network for Generative Commonsense Reasoning
• Exploring the Combination of Contextual Word Embeddings and Knowledge Graph Embeddings
• Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• DuTrust: A Sentiment Analysis Dataset for Trustworthiness Evaluation
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization
• Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models
• K-XLNet: A General Method for Combining Explicit Knowledge with Language Model Pretraining
• X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications
• Linguistically Informed Masking for Representation Learning in the Patent Domain
• Unsupervised Improvement of Factual Knowledge in Language Models
• Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension
• Retrieve Synonymous keywords for Frequent Queries in Sponsored Search in a Data Augmentation Way
• AT-BERT: Adversarial Training BERT for Acronym Identification Winning Solution for SDU@AAAI-21
• Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation
• An Understanding-Oriented Robust Machine Reading Comprehension Model
• Bert4XMR: Cross-Market Recommendation with Bidirectional Encoder Representations from Transformer
• Pre-trained Language Model based Ranking in Baidu Search
• K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce
• Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
• Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
• BERTnesia: Investigating the capture and forgetting of knowledge in BERT
• A Simple and Efficient Multi-Task Learning Approach for Conditioned Dialogue Generation
• DuReader_robust: A Chinese Dataset Towards Evaluating Robustness and Generalization of Machine Reading Comprehension in Real-World Applications
• Learning Better Sentence Representation with Syntax Information
• Probabilistically Masked Language Model Capable of Autoregressive Generation in Arbitrary Word Order
• Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
• Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation
• CLOP: Video-and-Language Pre-Training with Knowledge Regularizations
• Analysing the Effect of Masking Length Distribution of MLM: An Evaluation Framework and Case Study on Chinese MRC Datasets
• K-AID: Enhancing Pre-trained Language Models with Domain Knowledge for Question Answering
• Chinese Lexical Simplification
• Does Chinese BERT Encode Word Structure?
• KddRES: A Multi-level Knowledge-driven Dialogue Dataset for Restaurant Towards Customized Dialogue System
• Unlocking Efficiency: Adaptive Masking for Gene Transformer Models
• When Retriever-Reader Meets Scenario-Based Multiple-Choice Questions
• Data Efficient Masked Language Modeling for Vision and Language
• BIG MOOD: Relating Transformers to Explicit Commonsense Knowledge
• A Survey on Green Deep Learning
• Simple and Effective Input Reformulations for Translation
• EntEval: A Holistic Evaluation Benchmark for Entity Representations
• A Concept Knowledge-Driven Keywords Retrieval Framework for Sponsored Search
• Identify, Align, and Integrate: Matching Knowledge Graphs to Commonsense Reasoning Tasks
• Proactive Interaction Framework for Intelligent Social Receptionist Robots
• Towards Improving Interpretability of Language Model Generation through a Structured Knowledge Discovery Approach