DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

Abstract

This paper presents a new pre-trained language model, DeBERTaV3, which improves the original DeBERTa model by replacing mask language modeling (MLM) with replaced token detection (RTD), a more sample-efficient pre-training task. Our analysis shows that vanilla embedding sharing in ELECTRA hurts training efficiency and model performance. This is because the training losses of the discriminator and the generator pull token embeddings in different directions, creating the "tug-of-war" dynamics. We thus propose a new gradient-disentangled embedding sharing method that avoids the tug-of-war dynamics, improving both training efficiency and the quality of the pre-trained model. We have pre-trained DeBERTaV3 using the same settings as DeBERTa to demonstrate its exceptional performance on a wide range of downstream natural language understanding (NLU) tasks. Taking the GLUE benchmark with eight tasks as an example, the DeBERTaV3 Large model achieves a 91.37% average score, which is 1.37% over DeBERTa and 1.91% over ELECTRA, setting a new state-of-the-art (SOTA) among the models with a similar structure. Furthermore, we have pre-trained a multi-lingual model mDeBERTa and observed a larger improvement over strong baselines compared to English models. For example, the mDeBERTa Base achieves a 79.8% zero-shot cross-lingual accuracy on XNLI and a 3.6% improvement over XLM-R Base, creating a new SOTA on this benchmark. We have made our pre-trained models and inference code publicly available at https://github.com/microsoft/DeBERTa.

arXiv comment16 pages, 10 tables, 2 Figures. The DeBERTaV3 model significantly improves performance of the downstream NLU tasks over models with a similar structure, e.g. DeBERTaV3 large achieves 91.37% average GLUE score which is 1.37% over DeBERTa large. XSmall has only 22M backbone parameters, but significantly outperforms RoBERTa/XLNet-base. Paper is published as a conference paper at ICLR 2023

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Language Models are Few-Shot Learners
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• A Simple Method for Commonsense Reasoning
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers

Cited by in corpus (27)

• ChatGPT: Jack of all trades, master of none
• The Perils & Promises of Fact-checking with Large Language Models
• LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
• Flexible, Model-Agnostic Method for Materials Data Extraction from Text Using General Purpose Language Models
• Language Modelling with Pixels
• Findings of the The RuATD Shared Task 2022 on Artificial Text Detection in Russian
• Improving Bilingual Lexicon Induction with Cross-Encoder Reranking
• DIALOG-22 RuATD Generated Text Detection
• A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions
• IDIAPers @ Causal News Corpus 2022: Efficient Causal Relation Identification Through a Prompt-based Few-shot Approach
• Table-to-Text Generation with Pretrained Diffusion Models
• What distinguishes conspiracy from critical narratives? A computational analysis of oppositional discourse
• An Interpretable Ensemble of Graph and Language Models for Improving Search Relevance in E-Commerce
• A Cognitive Study on Semantic Similarity Analysis of Large Corpora: A Transformer-based Approach
• Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications
• Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Data
• RAMP: Retrieval and Attribute-Marking Enhanced Prompting for Attribute-Controlled Translation
• Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk
• ÚFAL CorPipe at CRAC 2023: Larger Context Improves Multilingual Coreference Resolution
• X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications
• End-to-End Argument Mining over Varying Rhetorical Structures
• Word Sense Linking: Disambiguating Outside the Sandbox
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data
• Data Augmentation Method Utilizing Template Sentences for Variable Definition Extraction
• Do "English" Named Entity Recognizers Work Well on Global Englishes?
• SAUCE: Summary Analysis Using Conversation Entailment