DeBERTa: Decoding-enhanced BERT with Disentangled Attention

Abstract

Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions, respectively. Second, an enhanced mask decoder is used to incorporate absolute positions in the decoding layer to predict the masked tokens in model pre-training. In addition, a new virtual adversarial training method is used for fine-tuning to improve models' generalization. We show that these techniques significantly improve the efficiency of model pre-training and the performance of both natural language understanding (NLU) and natural langauge generation (NLG) downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). Notably, we scale up DeBERTa by training a larger version that consists of 48 Transform layers with 1.5 billion parameters. The significant performance boost makes the single DeBERTa model surpass the human performance on the SuperGLUE benchmark (Wang et al., 2019a) for the first time in terms of macro-average score (89.9 versus 89.8), and the ensemble DeBERTa model sits atop the SuperGLUE leaderboard as of January 6, 2021, out performing the human baseline by a decent margin (90.3 versus 89.8).

20 pages,5 figures, 13 tables. In v2, we scale up DeBERTa to 1.5B parameters and it surpasses the human performance on SuperGLUE leaderboard for the first time as of December 29, 2020. In v3, we replace MLM with RTD objective which significantly improves the model performance

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ERNIE: Enhanced Representation through Knowledge Integration
• Generating Long Sequences with Sparse Transformers
• Pointer Sentinel Mixture Models
• Deep Learning Based Text Classification: A Comprehensive Review
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Reformer: The Efficient Transformer
• A Simple Method for Commonsense Reasoning
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• Adversarial Training for Large Neural Language Models
• X-SQL: reinforce schema representation with context
• Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
• A Hybrid Neural Network Model for Commonsense Reasoning

Cited by in corpus (69)

• LoRA: Low-Rank Adaptation of Large Language Models
• Evaluating Large Language Models Trained on Code
• Conditional Positional Encodings for Vision Transformers
• Measuring Mathematical Problem Solving With the MATH Dataset
• polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Pretrained Transformers for Text Ranking: BERT and Beyond
• CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation
• Backdoor Pre-trained Models Can Transfer to All
• MisRoBÆRTa: Transformers versus Misinformation
• Rethinking Positional Encoding in Language Pre-training
• The Perils & Promises of Fact-checking with Large Language Models
• M6: A Chinese Multimodal Pretrainer
• Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• SODA: A Natural Language Processing Package to Extract Social Determinants of Health for Cancer Studies
• LOREN: Logic-Regularized Reasoning for Interpretable Fact Verification
• Large-Scale Text Analysis Using Generative Language Models: A Case Study in Discovering Public Value Expressions in AI Patents
• Investigating the Limitations of Transformers with Simple Arithmetic Tasks
• A transformer based approach for fighting COVID-19 fake news
• Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
• Rethinking Positional Encoding
• Taming Sparsely Activated Transformer with Stochastic Experts
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection
• Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
• Relative Positional Encoding for Transformers with Linear Complexity
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
• WangchanBERTa: Pretraining transformer-based Thai Language Models
• Defending Against Backdoor Attacks in Natural Language Generation
• How to Query Language Models?
• Trusting RoBERTa over BERT: Insights from CheckListing the Natural Language Inference Task
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models
• A Study of Situational Reasoning for Traffic Understanding
• SheffieldVeraAI at SemEval-2023 Task 3: Mono and multilingual approaches for news genre, topic and persuasion technique classification
• Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining
• Topic-Guided Abstractive Text Summarization: a Joint Learning Approach
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• Direction is what you need: Improving Word Embedding Compression in Large Language Models
• Blacks is to Anger as Whites is to Joy? Understanding Latent Affective Bias in Large Pre-trained Neural Language Models
• $k$Folden: $k$-Fold Ensemble for Out-Of-Distribution Detection
• FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation
• CFEVER: A Chinese Fact Extraction and VERification Dataset
• Improving Controllability of Educational Question Generation by Keyword Provision
• Improving Self-supervised Pre-training via a Fully-Explored Masked Language Model
• Robustly Optimized and Distilled Training for Natural Language Understanding
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• Probabilistic Impact Score Generation using Ktrain-BERT to Identify Hate Words from Twitter Discussions
• Interpreting A Pre-trained Model Is A Key For Model Architecture Optimization: A Case Study On Wav2Vec 2.0
• Multiplicative Position-aware Transformer Models for Language Understanding
• Is text normalization relevant for classifying medieval charters?
• Limited Effectiveness of LLM-based Data Augmentation for COVID-19 Misinformation Stance Detection
• Positional Artefacts Propagate Through Masked Language Model Embeddings
• CodeQA: A Question Answering Dataset for Source Code Comprehension
• MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models
• Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
• Passage Retrieval of Polish Texts Using OKAPI BM25 and an Ensemble of Cross Encoders
• ARCH: Efficient Adversarial Regularized Training with Caching
• COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences
• Improving Patent Mining and Relevance Classification using Transformers
• Adversarial Regularization as Stackelberg Game: An Unrolled Optimization Approach
• STaCK: Sentence Ordering with Temporal Commonsense Knowledge
• Don't Go Far Off: An Empirical Study on Neural Poetry Translation
• The Stem Cell Hypothesis: Dilemma behind Multi-Task Learning with Transformer Encoders