Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
arXiv:2104.00676
Abstract
This work aims to empirically clarify a recently discovered perspective that label smoothing is incompatible with knowledge distillation. We begin by introducing the motivation behind on how this incompatibility is raised, i.e., label smoothing erases relative information between teacher logits. We provide a novel connection on how label smoothing affects distributions of semantically similar and dissimilar classes. Then we propose a metric to quantitatively measure the degree of erased information in sample's representation. After that, we study its one-sidedness and imperfection of the incompatibility view through massive analyses, visualizations and comprehensive experiments on Image Classification, Binary Networks, and Neural Machine Translation. Finally, we broadly discuss several circumstances wherein label smoothing will indeed lose its effectiveness. Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html.
arXiv commentICLR 2021. Project page: http://zhiqiangshen.com/projects/LS_and_KD/index.html
References in corpus (9)
- Distilling the Knowledge in a Neural Network
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Wide Residual Networks
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Label Refinery: Improving ImageNet Classification through Label Progression
- Multilingual Neural Machine Translation with Knowledge Distillation
- Understanding and Improving Knowledge Distillation
- Training Binary Neural Networks with Real-to-Binary Convolutions
- MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks