Self-Distillation as Instance-Specific Label Smoothing
arXiv:2006.05065
Abstract
It has been recently demonstrated that multi-generational self-distillation can improve generalization. Despite this intriguing observation, reasons for the enhancement remain poorly understood. In this paper, we first demonstrate experimentally that the improved performance of multi-generational self-distillation is in part associated with the increasing diversity in teacher predictions. With this in mind, we offer a new interpretation for teacher-student training as amortized MAP estimation, such that teacher predictions enable instance-specific regularization. Our framework allows us to theoretically relate self-distillation to label smoothing, a commonly used technique that regularizes predictive uncertainty, and suggests the importance of predictive diversity in addition to predictive uncertainty. We present experimental results using multiple datasets and neural network architectures that, overall, demonstrate the utility of predictive diversity. Finally, we propose a novel instance-specific label smoothing technique that promotes predictive diversity without the need for a separately trained teacher model. We provide an empirical evaluation of the proposed method, which, we find, often outperforms classical label smoothing.
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- mixup: Beyond Empirical Risk Minimization
- On Calibration of Modern Neural Networks
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Regularizing Neural Networks by Penalizing Confident Output Distributions
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- Bayesian Dark Knowledge
- Towards Understanding Knowledge Distillation
- Knowledge Transfer with Jacobian Matching
- Maximum-Entropy Fine-Grained Classification