Locally Linear Region Knowledge Distillation

Abstract

Knowledge distillation (KD) is an effective technique to transfer knowledge from one neural network (teacher) to another (student), thus improving the performance of the student. To make the student better mimic the behavior of the teacher, the existing work focuses on designing different criteria to align their logits or representations. Different from these efforts, we address knowledge distillation from a novel data perspective. We argue that transferring knowledge at sparse training data points cannot enable the student to well capture the local shape of the teacher function. To address this issue, we propose locally linear region knowledge distillation (${\mathrm{L}}^2$RKD) which transfers the knowledge in local, linear regions from a teacher to a student. This is achieved by enforcing the student to mimic the outputs of the teacher function in local, linear regions. To the end, the student is able to better capture the local shape of the teacher function and thus achieves a better performance. Despite its simplicity, extensive experiments demonstrate that ${\mathrm{L}}^2$RKD is superior to the original KD in many aspects as it outperforms KD and the other state-of-the-art approaches by a large margin, shows robustness and superiority under few-shot settings, and is more compatible with the existing distillation approaches to further improve their performances significantly.

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Knowledge Distillation by On-the-Fly Native Ensemble
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Large scale distributed neural network training through online distillation
• Knowledge Transfer with Jacobian Matching
• Feature-map-level Online Adversarial Knowledge Distillation
• Knowledge Distillation from Internal Representations
• Teacher-Student Compression with Generative Adversarial Networks