Revisiting Few-sample BERT Fine-tuning

Abstract

This paper is a study of fine-tuning of BERT contextual representations, with focus on commonly observed instabilities in few-sample scenarios. We identify several factors that cause this instability: the common use of a non-standard optimization method with biased gradient estimation; the limited applicability of significant parts of the BERT network for down-stream tasks; and the prevalent practice of using a pre-determined, and small number of training iterations. We empirically test the impact of these factors, and identify alternative practices that resolve the commonly observed instability of the process. In light of these observations, we re-visit recently proposed methods to improve few-sample fine-tuning with BERT and re-evaluate their effectiveness. Generally, we observe the impact of these methods diminishes significantly with our modified process.

arXiv commentCode available at https://github.com/asappresearch/revisit-bert-finetuning

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• How transferable are features in deep neural networks?
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• BERTScore: Evaluating Text Generation with BERT
• Frustratingly Easy Domain Adaptation
• REALM: Retrieval-Augmented Language Model Pre-Training
• Training Tips for the Transformer Model
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
• Incorporating BERT into Neural Machine Translation
• Parameter-Efficient Transfer Learning for NLP
• What do you learn from context? Probing for sentence structure in contextualized word representations
• Simple Applications of BERT for Ad Hoc Document Retrieval
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• The Microsoft Toolkit of Multi-Task Deep Neural Networks for Natural Language Understanding
• RIFLE: Backpropagation in Depth for Deep Transfer Learning through Re-Initializing the Fully-connected LayEr

Cited by in corpus (24)

• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
• Making Pre-trained Language Models Better Few-shot Learners
• Entailment as Few-Shot Learner
• On the Effect of Dropping Layers of Pre-trained Transformer Models
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning
• Variational Information Bottleneck for Effective Low-Resource Fine-Tuning
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• KNN-BERT: Fine-Tuning Pre-Trained Models with KNN Classifier
• Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
• Rethinking embedding coupling in pre-trained language models
• Adapting by Pruning: A Case Study on BERT
• An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
• To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment
• Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge
• On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
• How Does Adversarial Fine-Tuning Benefit BERT?
• Leveraging Time Irreversibility with Order-Contrastive Pre-training
• LG4AV: Combining Language Models and Graph Neural Networks for Author Verification
• Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization
• Robust Document Representations using Latent Topics and Metadata
• MDQE: A More Accurate Direct Pretraining for Machine Translation Quality Estimation