What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning

Abstract

Pretrained transformer-based language models have achieved state of the art across countless tasks in natural language processing. These models are highly expressive, comprising at least a hundred million parameters and a dozen layers. Recent evidence suggests that only a few of the final layers need to be fine-tuned for high quality on downstream tasks. Naturally, a subsequent research question is, "how many of the last layers do we need to fine-tune?" In this paper, we precisely answer this question. We examine two recent pretrained language models, BERT and RoBERTa, across standard tasks in textual entailment, semantic similarity, sentiment analysis, and linguistic acceptability. We vary the number of final layers that are fine-tuned, then study the resulting change in task-specific effectiveness. We show that only a fourth of the final layers need to be fine-tuned to achieve 90% of the original quality. Surprisingly, we also find that fine-tuning all layers does not always help.

arXiv comment5 pages

References in corpus (4)

• Adam: A Method for Stochastic Optimization
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Cited by in corpus (8)

• Pretrained Transformers for Text Ranking: BERT and Beyond
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
• FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging
• Language Scaling for Universal Suggested Replies Model
• A Conditional Generative Matching Model for Multi-lingual Reply Suggestion
• A System for Worldwide COVID-19 Information Aggregation