Following the Teacher's Footsteps: Scheduled Checkpoint Distillation for Domain-Specific LLMs

Cheng Feng, Chaoliang Zhong, Jun Sun +1

Large language models (LLMs) are challenging to deploy for domain-specific tasks due to their massive scale. While distilling a fine-tuned LLM into a smaller student model is a pro…