A Dual-Space Framework for General Knowledge Distillation of Large Language Models

Xue Zhang, Songming Zhang, Yunlong Liang +4

Knowledge distillation (KD) is a promising solution to compress large language models (LLMs) by transferring their knowledge to smaller models. During this process, white-box KD me…