Revisiting Knowledge Distillation: The Hidden Role of Dataset Size

Giulia Lanzillotta, Felix Sarnthein, Gil Kur +2

The concept of knowledge distillation (KD) describes the training of a student model from a teacher model and is a widely adopted technique in deep learning. However, it is still n…