Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping

Abstract

Fine-tuning pretrained contextual word embedding models to supervised downstream tasks has become commonplace in natural language processing. This process, however, is often brittle: even with the same hyperparameter values, distinct random seeds can lead to substantially different results. To better understand this phenomenon, we experiment with four datasets from the GLUE benchmark, fine-tuning BERT hundreds of times on each while varying only the random seeds. We find substantial performance increases compared to previously reported results, and we quantify how the performance of the best-found model varies as a function of the number of fine-tuning trials. Further, we examine two factors influenced by the choice of random seed: weight initialization and training data order. We find that both contribute comparably to the variance of out-of-sample performance, and that some weight initializations perform well across all tasks explored. On small datasets, we observe that many fine-tuning trials diverge part of the way through training, and we offer best practices for practitioners to stop training less promising runs early. We publicly release all of our experimental data, including training and validation scores for 2,100 trials, to encourage further analysis of training dynamics during fine-tuning.

References in corpus (2)

• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks

Cited by in corpus (73)

• Pre-trained Models for Natural Language Processing: A Survey
• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• Revisiting Few-sample BERT Fine-tuning
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• True Few-Shot Learning with Language Models
• Making Pre-trained Language Models Better Few-shot Learners
• Carbon Emissions and Large Neural Network Training
• CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Variational Information Bottleneck for Effective Low-Resource Fine-Tuning
• Learning to summarize from human feedback
• When BERT Plays the Lottery, All Tickets Are Winning
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• PTR: Prompt Tuning with Rules for Text Classification
• LMMS Reloaded: Transformer-based Sense Embeddings for Disambiguation and Beyond
• BERT & Family Eat Word Salad: Experiments with Text Understanding
• CPM: A Large-scale Generative Chinese Pre-trained Language Model
• Better Fine-Tuning by Reducing Representational Collapse
• The MultiBERTs: BERT Reproductions for Robustness Analysis
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Addressing machine learning concept drift reveals declining vaccine sentiment during the COVID-19 pandemic
• Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
• I-BERT: Integer-only BERT Quantization
• On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation
• Transferring Inductive Biases through Knowledge Distillation
• KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier
• Detecting Frames in News Headlines and Lead Images in U.S. Gun Violence Coverage
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• DQI: Measuring Data Quality in NLP
• TextHide: Tackling Data Privacy in Language Understanding Tasks
• Cold-start Active Learning through Self-supervised Language Modeling
• The Out-of-Distribution Problem in Explainability and Search Methods for Feature Importance Explanations
• NLPDove at SemEval-2020 Task 12: Improving Offensive Language Detection with Cross-lingual Transfer
• An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Models
• Sentiment analysis in tweets: an assessment study from classical to modern text representation models
• Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach
• Leakage-Adjusted Simulatability: Can Models Generate Non-Trivial Explanations of Their Behavior in Natural Language?
• Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning
• On the Transformer Growth for Progressive BERT Training
• Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level
• Linking Common Vulnerabilities and Exposures to the MITRE ATT&CK Framework: A Self-Distillation Approach
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge
• Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
• Language Model Evaluation in Open-ended Text Generation
• Learn Faster and Forget Slower via Fast and Stable Task Adaptation
• Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity
• The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures
• UoB at SemEval-2021 Task 5: Extending Pre-Trained Language Models to Include Task and Domain-Specific Information for Toxic Span Prediction
• SkullEngine: A Multi-stage CNN Framework for Collaborative CBCT Image Segmentation and Landmark Detection
• Exploiting Token and Path-based Representations of Code for Identifying Security-Relevant Commits
• Bi-Granularity Contrastive Learning for Post-Training in Few-Shot Scene
• ReINTEL Challenge 2020: Exploiting Transfer Learning Models for Reliable Intelligence Identification on Vietnamese Social Network Sites
• AMU-EURANOVA at CASE 2021 Task 1: Assessing the stability of multilingual BERT
• Noise Stability Regularization for Improving BERT Fine-tuning
• Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning
• How much pretraining data do language models need to learn syntax?
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• GiBERT: Introducing Linguistic Knowledge into BERT through a Lightweight Gated Injection Method
• Pre-Training Transformers as Energy-Based Cloze Models
• WASSA@IITK at WASSA 2021: Multi-task Learning and Transformer Finetuning for Emotion Classification and Empathy Prediction
• Combining Deep Generative Models and Multi-lingual Pretraining for Semi-supervised Document Classification
• Structural analysis of an all-purpose question answering model
• Beyond Fine-tuning: Few-Sample Sentence Embedding Transfer
• Double Descent Optimization Pattern and Aliasing: Caveats of Noisy Labels