Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
arXiv:2010.10090
Abstract
Knowledge distillation is a strategy of training a student network with guide of the soft output from a teacher network. It has been a successful method of model compression and knowledge transfer. However, currently knowledge distillation lacks a convincing theoretical understanding. On the other hand, recent finding on neural tangent kernel enables us to approximate a wide neural network with a linear model of the network's random features. In this paper, we theoretically analyze the knowledge distillation of a wide neural network. First we provide a transfer risk bound for the linearized model of the network. Then we propose a metric of the task's training difficulty, called data inefficiency. Based on this metric, we show that for a perfect teacher, a high ratio of teacher's soft labels can be beneficial. Finally, for the case of imperfect teacher, we find that hard labels can correct teacher's wrong prediction, which explains the practice of mixing hard and soft labels.
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- Risk bounds for statistical learning
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Towards Understanding Knowledge Distillation
- The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
- Understanding and Improving Knowledge Distillation
- Generalization Error Bounds of Gradient Descent for Learning Over-parameterized Deep ReLU Networks
- Graph-based Knowledge Distillation by Multi-head Attention Network
- Distillation Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network