Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates

Parjanya Prajakta Prashant, Jiongli Zhu, Aldan Creo +1

Fine-tuning large language models on new data improves task performance but degrades capabilities learned during pretraining, a phenomenon known as catastrophic forgetting. Existin…