Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation

Bowei He, Yankai Chen, Xiaokun Zhang +4

Knowledge distillation from Large Language Models (LLMs) to smaller models has emerged as a critical technique for deploying efficient AI systems. However, current methods for dist…