ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

Abstract

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

References in corpus (37)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Scaling Laws for Neural Language Models
• Zero-Shot Text-to-Image Generation
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ERNIE: Enhanced Representation through Knowledge Integration
• End-to-End Neural Ad-hoc Ranking with Kernel Pooling
• A large annotated corpus for learning natural language inference
• A Simple Method for Commonsense Reasoning
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction
• XNLI: Evaluating Cross-lingual Sentence Representations
• Glyce: Glyph-vectors for Chinese Character Representations
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• ERNIE: Enhanced Language Representation with Informative Entities
• PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
• DRCD: a Chinese Machine Reading Comprehension Dataset
• Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering
• CLUE: A Chinese Language Understanding Evaluation Benchmark
• DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
• M6: A Chinese Multimodal Pretrainer
• Integrating Graph Contextualized Knowledge into Pre-trained Language Models
• A Discourse-Level Named Entity Recognition and Relation Extraction Dataset for Chinese Literature Text
• ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders
• SKEP: Sentiment Knowledge Enhanced Pre-training for Sentiment Analysis
• Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense
• Knowledge Enhanced Contextual Word Representations
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• Long and Diverse Text Generation with Planning-based Hierarchical Variational Model
• ERNIE-Gram: Pre-Training with Explicitly N-Gram Masked Language Modeling for Natural Language Understanding
• ChID: A Large-scale Chinese IDiom Dataset for Cloze Test
• MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization
• Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge

Cited by in corpus (14)

• Unifying Large Language Models and Knowledge Graphs: A Roadmap
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation
• On Transferability of Prompt Tuning for Natural Language Processing
• Bias Against 93 Stigmatized Groups in Masked Language Models and Downstream Sentiment Classification Tasks
• A Survey on Symbolic Knowledge Distillation of Large Language Models
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
• Iterative self-transfer learning: A general methodology for response time-history prediction based on small dataset
• PLATO-XL: Exploring the Large-scale Pre-training of Dialogue Generation
• The Dark Side of the Language: Pre-trained Transformers in the DarkNet
• Argumentative Text Generation in Economic Domain
• X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications
• MvSR-NAT: Multi-view Subset Regularization for Non-Autoregressive Machine Translation