Improve Knowledge Distillation via Label Revision and Data Selection

Weichao Lan, Yiu-ming Cheung, Qing Xu +4

Knowledge distillation (KD) has become a widely used technique in the field of model compression, which aims to transfer knowledge from a large teacher model to a lightweight stude…