Quantification of Large Language Model Distillation

Sunbowen Lee, Junting Zhou, Chang Ao +11

Model distillation is a fundamental technique in building large language models (LLMs), transferring knowledge from a teacher model to a student model. However, distillation can le…