Analysis of Knowledge Transfer in Kernel Regime

Abstract

Knowledge transfer is shown to be a very successful technique for training neural classifiers: together with the ground truth data, it uses the "privileged information" (PI) obtained by a "teacher" network to train a "student" network. It has been observed that classifiers learn much faster and more reliably via knowledge transfer. However, there has been little or no theoretical analysis of this phenomenon. To bridge this gap, we propose to approach the problem of knowledge transfer by regularizing the fit between the teacher and the student with PI provided by the teacher. Using tools from dynamical systems theory, we show that when the student is an extremely wide two layer network, we can analyze it in the kernel regime and show that it is able to interpolate between PI and the given data. This characterization sheds new light on the relation between the training error and capacity of the student relative to the teacher. Another contribution of the paper is a quantitative statement on the convergence of student network. We prove that the teacher reduces the number of required iterations for a student to learn, and consequently improves the generalization power of the student. We give corresponding experimental analysis that validates the theoretical results and yield additional insights.

The work is published by CIKM 2022

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Towards Understanding Knowledge Distillation
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Width Provably Matters in Optimization for Deep Linear Neural Networks