Showing 2024Show all
2 papers · 1 filter
cs.CL2024
LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
Runming Yang, Taiqiang Wu, Jiahao Wang +4
Knowledge distillation (KD) has been a predominant method for compressing Large Language Models (LLMs). In this paper, we first revisit KD and Low-Rank Adaption (LoRA) and demonstr…
cs.CL2024
LoCa: Logit Calibration for Knowledge Distillation
Runming Yang, Taiqiang Wu, Yujiu Yang
Knowledge Distillation (KD), aiming to train a better student model by mimicking the teacher model, plays an important role in model compression. One typical way is to align the ou…