TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models

Yushi Guan, Jeanine Ohene-Agyei, Daniel Kwan +3

To embed domain-specific or specialized knowledge into pre-trained foundation models, fine-tuning using techniques such as parameter efficient fine-tuning (e.g. LoRA) is a common p…