Revisiting Knowledge Distillation via Label Smoothing Regularization

Abstract

Knowledge Distillation (KD) aims to distill the knowledge of a cumbersome teacher model into a lightweight student model. Its success is generally attributed to the privileged information on similarities among categories provided by the teacher model, and in this sense, only strong teacher models are deployed to teach weaker students in practice. In this work, we challenge this common belief by following experimental observations: 1) beyond the acknowledgment that the teacher can improve the student, the student can also enhance the teacher significantly by reversing the KD procedure; 2) a poorly-trained teacher with much lower accuracy than the student can still improve the latter significantly. To explain these observations, we provide a theoretical analysis of the relationships between KD and label smoothing regularization. We prove that 1) KD is a type of learned label smoothing regularization and 2) label smoothing regularization provides a virtual teacher model for KD. From these results, we argue that the success of KD is not fully due to the similarity information between categories from teachers, but also to the regularization of soft targets, which is equally or even more important. Based on these analyses, we further propose a novel Teacher-free Knowledge Distillation (Tf-KD) framework, where a student model learns from itself or manuallydesigned regularization distribution. The Tf-KD achieves comparable performance with normal KD from a superior teacher, which is well applied when a stronger teacher model is unavailable. Meanwhile, Tf-KD is generic and can be directly deployed for training deep neural networks. Without any extra computation cost, Tf-KD achieves up to 0.65\% improvement on ImageNet over well-established baseline models, which is superior to label smoothing regularization.

arXiv commentCVPR2020 Oral, codes: https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• When Does Label Smoothing Help?
• Born Again Neural Networks
• Large scale distributed neural network training through online distillation

Cited by in corpus (12)

• Knowledge Distillation: A Survey
• Understanding and Improving Knowledge Distillation
• Towards Understanding Label Smoothing
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Transferring Inductive Biases through Knowledge Distillation
• Soft Labeling Affects Out-of-Distribution Detection of Deep Neural Networks
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery
• Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation
• Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System
• Prime-Aware Adaptive Distillation