Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs

Ruihan Jin, Pengpeng Shao, Zhengqi Wen +5

Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditio…