Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs

Anshumann, Mohd Abbas Zaidi, Akhil Kedia +5

Knowledge distillation can be a cost-effective technique to distill knowledge in Large Language Models, if the teacher output logits can be pre-computed and cached. However, succes…