ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

Abstract

Recently, pre-trained models have achieved state-of-the-art results in various language understanding tasks, which indicates that pre-training on large-scale corpora may play a crucial role in natural language processing. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entity, semantic closeness and discourse relations. In order to extract to the fullest extent, the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which builds and learns incrementally pre-training tasks through constant multi-task learning. Experimental results demonstrate that ERNIE 2.0 outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several common tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

11 pages, 3 figures and 7 tables; Accepted by AAAI 2020

References in corpus (8)

• Efficient Estimation of Word Representations in Vector Space
• Cross-lingual Language Model Pretraining
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• ERNIE: Enhanced Representation through Knowledge Integration
• A large annotated corpus for learning natural language inference
• Multi-Task Deep Neural Networks for Natural Language Understanding
• XNLI: Evaluating Cross-lingual Sentence Representations
• DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications

Cited by in corpus (27)

• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Adversarial Training for Large Neural Language Models
• CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese
• Leveraging Contextual Embeddings for Detecting Diachronic Semantic Shift
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
• RealFormer: Transformer Likes Residual Attention
• Pre-Trained Models: Past, Present and Future
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• Response Generation with Context-Aware Prompt Learning
• Continual BERT: Continual Learning for Adaptive Extractive Summarization of COVID-19 Literature
• Training Multilingual Pre-trained Language Model with Byte-level Subwords
• Alzheimer's Disease Detection from Spontaneous Speech through Combining Linguistic Complexity and (Dis)Fluency Features with Pretrained Language Models
• BiQGEMM: Matrix Multiplication with Lookup Table For Binary-Coding-based Quantized DNNs
• Enhancing Answer Boundary Detection for Multilingual Machine Reading Comprehension
• Measuring Patent Claim Generation by Span Relevancy
• DLGNet: A Transformer-based Model for Dialogue Response Generation
• Application of Pre-training Models in Named Entity Recognition
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• kk2018 at SemEval-2020 Task 9: Adversarial Training for Code-Mixing Sentiment Classification
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression
• BET: A Backtranslation Approach for Easy Data Augmentation in Transformer-based Paraphrase Identification Context
• Understanding Semantics from Speech Through Pre-training
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Improving QA Generalization by Concurrent Modeling of Multiple Biases
• Self-supervised Regularization for Text Classification