Knowledge Distillation with Deep Supervision

Shiya Luo, Defang Chen, Can Wang

Knowledge distillation aims to enhance the performance of a lightweight student model by exploiting the knowledge from a pre-trained cumbersome teacher model. However, in the tradi…