Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation

Peter Belcak, Greg Heinrich, Jan Kautz +1

Finetuning language models for a new domain inevitably leads to the deterioration of their general performance. This becomes more pronounced the more limited the finetuning data re…