Expandable Residual Approximation for Knowledge Distillation

Zhaoyi Yan, Binghui Chen, Yunfan Liu +1

Knowledge distillation (KD) aims to transfer knowledge from a large-scale teacher model to a lightweight one, significantly reducing computational and storage requirements. However…