Measuring the Instability of Fine-Tuning

Yupei Du, Dong Nguyen

Fine-tuning pre-trained language models on downstream tasks with varying random seeds has been shown to be unstable, especially on small datasets. Many previous studies have invest…