Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization

Junlin He, Yihong Tang, Tong Nie +5

Efficient Distillation (EDistill) compresses large language models (LLMs) by structured pruning parameters and tuning lightweight modules with high training efficiency. Although th…