Self-Distillation Amplifies Regularization in Hilbert Space

Abstract

Knowledge distillation introduced in the deep learning context is a method to transfer knowledge from one architecture to another. In particular, when the architectures are identical, this is called self-distillation. The idea is to feed in predictions of the trained model as new target values for retraining (and iterate this loop possibly a few times). It has been empirically observed that the self-distilled model often achieves higher accuracy on held out data. Why this happens, however, has been a mystery: the self-distillation dynamics does not receive any new information about the task and solely evolves by looping over training. To the best of our knowledge, there is no rigorous understanding of this phenomenon. This work provides the first theoretical analysis of self-distillation. We focus on fitting a nonlinear function to training data, where the model space is Hilbert space and fitting is subject to ${\ell }_2$ regularization in this function space. We show that self-distillation iterations modify regularization by progressively limiting the number of basis functions that can be used to represent the solution. This implies (as we also verify empirically) that while a few rounds of self-distillation may reduce over-fitting, further rounds may lead to under-fitting and thus worse performance.

References in corpus (16)

• Distilling the Knowledge in a Neural Network
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Knowledge Distillation by On-the-Fly Native Ensemble
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• On Lazy Training in Differentiable Programming
• Bayesian Dark Knowledge
• Towards Understanding Knowledge Distillation
• Knowledge Transfer with Jacobian Matching
• Zero-Shot Knowledge Distillation in Deep Networks
• Hydra: Preserving Ensemble Diversity for Model Distillation
• Neural Kernels Without Tangents
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Why distillation helps: a statistical perspective
• Transferring Inductive Biases through Knowledge Distillation
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks

Cited by in corpus (17)

• Knowledge Distillation: A Survey
• Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
• Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Transferring Inductive Biases through Knowledge Distillation
• Noisy Self-Knowledge Distillation for Text Summarization
• Run Away From your Teacher: Understanding BYOL by a Novel Self-Supervised Approach
• Iterative Graph Self-Distillation
• Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learning
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations
• Churn Reduction via Distillation
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging