Distilling Knowledge via Intermediate Classifiers

Aryan Asadian, Amirali Salehi-Abari

The crux of knowledge distillation is to effectively train a resource-limited student model with the guide of a pre-trained larger teacher model. However, when there is a large dif…