Knowledge Distillation Layer that Lets the Student Decide

Ada Gorgun, Yeti Z. Gurbuz, A. Aydin Alatan

Typical technique in knowledge distillation (KD) is regularizing the learning of a limited capacity model (student) by pushing its responses to match a powerful model's (teacher).…