Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid it

Irene Trigueros-Lorca, Leonardo Concepción, Christian Wagner +2

The growing size of Convolutional Neural Networks has led to increasingly large and costly models. Knowledge Distillation (KD) addresses this by transferring knowledge from a large…