On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines

Abstract

Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: training the same model with multiple random seeds can result in a large variance of the task performance. Previous literature (Devlin et al., 2019; Lee et al., 2020; Dodge et al., 2020) identified two potential reasons for the observed instability: catastrophic forgetting and small size of the fine-tuning datasets. In this paper, we show that both hypotheses fail to explain the fine-tuning instability. We analyze BERT, RoBERTa, and ALBERT, fine-tuned on commonly used datasets from the GLUE benchmark, and show that the observed instability is caused by optimization difficulties that lead to vanishing gradients. Additionally, we show that the remaining variance of the downstream task performance can be attributed to differences in generalization where fine-tuned models with the same training loss exhibit noticeably different test performance. Based on our analysis, we present a simple but strong baseline that makes fine-tuning BERT-based models significantly more stable than the previously proposed approaches. Code to reproduce our results is available online: https://github.com/uds-lsv/bert-stable-fine-tuning.

arXiv commentICLR 2021

References in corpus (11)

• Decoupled Weight Decay Regularization
• Overcoming catastrophic forgetting in neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Fast is better than free: Revisiting adversarial training
• Pointer Sentinel Mixture Models
• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• Revisiting Few-sample BERT Fine-tuning
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?

Cited by in corpus (26)

• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Revisiting Few-sample BERT Fine-tuning
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality
• Variational Information Bottleneck for Effective Low-Resource Fine-Tuning
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Addressing machine learning concept drift reveals declining vaccine sentiment during the COVID-19 pandemic
• Small-Text: Active Learning for Text Classification in Python
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives
• Detecting Frames in News Headlines and Lead Images in U.S. Gun Violence Coverage
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Topic-Guided Sampling For Data-Efficient Multi-Domain Stance Detection
• Adapting an ASR Foundation Model for Spoken Language Assessment
• Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
• KInITVeraAI at SemEval-2023 Task 3: Simple yet Powerful Multilingual Fine-Tuning for Persuasion Techniques Detection
• $Q_{bias}$ -- A Dataset on Media Bias in Search Queries and Query Suggestions
• Linking Common Vulnerabilities and Exposures to the MITRE ATT&CK Framework: A Self-Distillation Approach
• Detecting Inappropriate Messages on Sensitive Topics that Could Harm a Company's Reputation
• How Does Adversarial Fine-Tuning Benefit BERT?
• GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
• Robust Transfer Learning with Pretrained Language Models through Adapters
• UU-Tax at SemEval-2022 Task 3: Improving the generalizability of language models for taxonomy classification through data augmentation
• Understanding Emotion Valence is a Joint Deep Learning Task
• Unlocking Efficiency: Adaptive Masking for Gene Transformer Models