Towards Understanding Knowledge Distillation

Abstract

Knowledge distillation, i.e., one classifier being trained on the outputs of another classifier, is an empirically very successful technique for knowledge transfer between classifiers. It has even been observed that classifiers learn much faster and more reliably if trained with the outputs of another classifier as soft labels, instead of from ground truth data. So far, however, there is no satisfactory theoretical explanation of this phenomenon. In this work, we provide the first insights into the working mechanisms of distillation by studying the special case of linear and deep linear classifiers. Specifically, we prove a generalization bound that establishes fast convergence of the expected risk of a distillation-trained linear classifier. From the bound and its proof we extract three key factors that determine the success of distillation: * data geometry -- geometric properties of the data distribution, in particular class separation, has a direct influence on the convergence speed of the risk; * optimization bias -- gradient descent optimization finds a very favorable minimum of the distillation objective; and * strong monotonicity -- the expected risk of the student classifier always decreases when the size of the training set grows.

arXiv commentICML'19. Post-edited to add related work. arXiv admin note: text overlap with arXiv:2003.13438 by other authors

Cited by in corpus (48)

• Knowledge Distillation: A Survey
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• Self-Distillation Amplifies Regularization in Hilbert Space
• Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?
• Understanding and Improving Knowledge Distillation
• Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions
• A Survey on Deep Neural Network Compression: Challenges, Overview, and Solutions
• Self-Distillation as Instance-Specific Label Smoothing
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• Hydra: Preserving Ensemble Diversity for Model Distillation
• UniTE: Unified Translation Evaluation
• Sub-Band Knowledge Distillation Framework for Speech Enhancement
• COVID-MobileXpert: On-Device COVID-19 Patient Triage and Follow-up using Chest X-rays
• MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection
• Why distillation helps: a statistical perspective
• Explaining Knowledge Distillation by Quantifying the Knowledge
• Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model
• Transferring Inductive Biases through Knowledge Distillation
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation
• Noisy Self-Knowledge Distillation for Text Summarization
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Distilling EEG Representations via Capsules for Affective Computing
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• Knowledge Distillation for Multi-task Learning
• Knowledge Distillation as Semiparametric Inference
• Dynamic Group Convolution for Accelerating Convolutional Neural Networks
• Search to Distill: Pearls are Everywhere but not the Eyes
• Compact representations of convolutional neural networks via weight pruning and quantization
• Robustness and Diversity Seeking Data-Free Knowledge Distillation
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Analysis of Knowledge Transfer in Kernel Regime
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• Deeplite Neutrino: An End-to-End Framework for Constrained Deep Learning Model Optimization
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• Temporal Self-Ensembling Teacher for Semi-Supervised Object Detection
• Follow Your Path: a Progressive Method for Knowledge Distillation
• Solvable Model for Inheriting the Regularization through Knowledge Distillation
• Churn Reduction via Distillation
• A Peek Into the Reasoning of Neural Networks: Interpreting with Structural Visual Concepts
• Transfer Heterogeneous Knowledge Among Peer-to-Peer Teammates: A Model Distillation Approach
• Adjoined Networks: A Training Paradigm with Applications to Network Compression
• Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks
• A Study of Non-autoregressive Model for Sequence Generation
• Tensor train decompositions on recurrent networks
• Lifelong Twin Generative Adversarial Networks
• Student-Teacher Learning from Clean Inputs to Noisy Inputs