Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network

Abstract

Distillation is a method to transfer knowledge from one model to another and often achieves higher accuracy with the same capacity. In this paper, we aim to provide a theoretical understanding on what mainly helps with the distillation. Our answer is "early stopping". Assuming that the teacher network is overparameterized, we argue that the teacher network is essentially harvesting dark knowledge from the data via early stopping. This can be justified by a new concept, {Anisotropic Information Retrieval (AIR)}, which means that the neural network tends to fit the informative information first and the non-informative information (including noise) later. Motivated by the recent development on theoretically analyzing overparameterized neural networks, we can characterize AIR by the eigenspace of the Neural Tangent Kernel(NTK). AIR facilities a new understanding of distillation. With that, we further utilize distillation to refine noisy labels. We propose a self-distillation algorithm to sequentially distill knowledge from the network in the previous training epoch to avoid memorizing the wrong labels. We also demonstrate, both theoretically and empirically, that self-distillation can benefit from more than just early stopping. Theoretically, we prove convergence of the proposed algorithm to the ground truth labels for randomly initialized overparameterized neural networks in terms of ${\ell }_2$ distance, while the previous result was on convergence in $0$-$1$ loss. The theoretical result ensures the learned neural network enjoy a margin on the training data which leads to better generalization. Empirically, we achieve better testing accuracy and entirely avoid early stopping which makes the algorithm more user-friendly.

Accepted by NeurIPS 2019 Workshop on Machine Learning with Guarantees. Submitted to other places

References in corpus (25)

• Distilling the Knowledge in a Neural Network
• Wide Residual Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Understanding deep learning requires rethinking generalization
• A Convergence Theory for Deep Learning via Over-Parameterization
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels
• Born Again Neural Networks
• Deep Learning is Robust to Massive Label Noise
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Dimensionality-Driven Learning with Noisy Labels
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• How does Disagreement Help Generalization against Label Corruption?
• Label Refinery: Improving ImageNet Classification through Label Progression
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Learning From Noisy Singly-labeled Data
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• A Priori Estimates of the Population Risk for Residual Networks
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Wasserstein Adversarial Regularization (WAR) on label noise

Cited by in corpus (15)

• Self-Distillation Amplifies Regularization in Hilbert Space
• Self-Adaptive Training: beyond Empirical Risk Minimization
• Why distillation helps: a statistical perspective
• Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise
• When Does Preconditioning Help or Hurt Generalization?
• Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
• Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• Selective Knowledge Distillation for Neural Machine Translation
• Churn Reduction via Distillation
• Pro-KD: Progressive Distillation by Following the Footsteps of the Teacher
• Sign-RIP: A Robust Restricted Isometry Property for Low-rank Matrix Recovery
• Knowledge Distillation with Noisy Labels for Natural Language Understanding
• Distilling Double Descent