Intelligent Learning Rate Distribution to reduce Catastrophic Forgetting in Transformers

Philip Kenneweg, Alexander Schulz, Sarah Schröder +1

Pretraining language models on large text corpora is a common practice in natural language processing. Fine-tuning of these models is then performed to achieve the best results on…