Solvable Model for Inheriting the Regularization through Knowledge Distillation
arXiv:2012.00194
Abstract
In recent years the empirical success of transfer learning with neural networks has stimulated an increasing interest in obtaining a theoretical understanding of its core properties. Knowledge distillation where a smaller neural network is trained using the outputs of a larger neural network is a particularly interesting case of transfer learning. In the present work, we introduce a statistical physics framework that allows an analytic characterization of the properties of knowledge distillation (KD) in shallow neural networks. Focusing the analysis on a solvable model that exhibits a non-trivial generalization gap, we investigate the effectiveness of KD. We are able to show that, through KD, the regularization properties of the larger teacher model can be inherited by the smaller student and that the yielded generalization performance is closely linked to and limited by the optimality of the teacher. Finally, we analyze the double descent phenomenology that can arise in the considered KD setting.
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- A modern maximum-likelihood theory for high-dimensional logistic regression
- The generalization error of random features regression: Precise asymptotics and double descent curve
- Phase diagram of glassy systems in an external field
- Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
- Towards Understanding Knowledge Distillation
- Understanding and Improving Knowledge Distillation
- Entropy landscape of solutions in the binary perceptron problem
- Learning may need only a few bits of synaptic precision
- On the role of synaptic stochasticity in training low-precision neural networks