Understanding and Improving Knowledge Distillation

Abstract

Knowledge Distillation (KD) is a model-agnostic technique to improve model quality while having a fixed capacity budget. It is a commonly used technique for model compression, where a larger capacity teacher model with better quality is used to train a more compact student model with better inference efficiency. Through distillation, one hopes to benefit from student's compactness, without sacrificing too much on model quality. Despite the large success of knowledge distillation, better understanding of how it benefits student model's training dynamics remains under-explored. In this paper, we categorize teacher's knowledge into three hierarchical levels and study its effects on knowledge distillation: (1) knowledge of the `universe', where KD brings a regularization effect through label smoothing; (2) domain knowledge, where teacher injects class relationships prior to student's logit layer geometry; and (3) instance specific knowledge, where teacher rescales student model's per-instance gradients based on its measurement on the event difficulty. Using systematic analyses and extensive empirical studies on both synthetic and real-world datasets, we confirm that the aforementioned three factors play a major role in knowledge distillation. Furthermore, based on our findings, we diagnose some of the failure cases of applying KD from recent studies.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• When Does Label Smoothing Help?
• Regularizing and Optimizing LSTM Language Models
• Born Again Neural Networks
• Billion-scale semi-supervised learning for image classification
• Scalable Private Learning with PATE
• Unifying distillation and privileged information
• Large scale distributed neural network training through online distillation
• Towards Understanding Knowledge Distillation
• Sequence-Level Knowledge Distillation
• Heated-Up Softmax Embedding
• Revisiting Knowledge Distillation via Label Smoothing Regularization

Cited by in corpus (24)

• Knowledge Distillation: A Survey
• MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
• Understanding and Improving Lexical Choice in Non-Autoregressive Translation
• Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation
• Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study
• Tackling the Unannotated: Scene Graph Generation with Bias-Reduced Models
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Why distillation helps: a statistical perspective
• Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation
• Transferring Inductive Biases through Knowledge Distillation
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation
• Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher
• Anti-Distillation: Improving reproducibility of deep networks
• Teacher's pet: understanding and mitigating biases in distillation
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
• Knowledge Distillation as Semiparametric Inference
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Periocular Embedding Learning with Consistent Knowledge Distillation from Face
• On the Demystification of Knowledge Distillation: A Residual Network Perspective
• Advances and Challenges in Deep Lip Reading
• Solvable Model for Inheriting the Regularization through Knowledge Distillation
• Understanding the Logit Distributions of Adversarially-Trained Deep Neural Networks
• Deploying a BERT-based Query-Title Relevance Classifier in a Production System: a View from the Trenches
• Distilling Double Descent