Explaining Knowledge Distillation by Quantifying the Knowledge

Abstract

This paper presents a method to interpret the success of knowledge distillation by quantifying and analyzing task-relevant and task-irrelevant visual concepts that are encoded in intermediate layers of a deep neural network (DNN). More specifically, three hypotheses are proposed as follows. 1. Knowledge distillation makes the DNN learn more visual concepts than learning from raw data. 2. Knowledge distillation ensures that the DNN is prone to learning various visual concepts simultaneously. Whereas, in the scenario of learning from raw data, the DNN learns visual concepts sequentially. 3. Knowledge distillation yields more stable optimization directions than learning from raw data. Accordingly, we design three types of mathematical metrics to evaluate feature representations of the DNN. In experiments, we diagnosed various DNNs, and above hypotheses were verified.

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• FitNets: Hints for Thin Deep Nets
• Understanding Neural Networks Through Deep Visualization
• Opening the Black Box of Deep Neural Networks via Information
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• Born Again Neural Networks
• Similarity of Neural Network Representations Revisited
• Learning how to explain neural networks: PatternNet and PatternAttribution
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• Information-theoretic analysis of generalization capability of learning algorithms
• Evaluating the Robustness of Neural Networks: An Extreme Value Theory Approach
• Towards Understanding Knowledge Distillation
• i-RevNet: Deep Invertible Networks
• Stiffness: A New Perspective on Generalization in Neural Networks
• Understanding training and generalization in deep learning by Fourier analysis
• Knowledge Consistency between Neural Networks and Beyond

Cited by in corpus (3)

• Knowledge Distillation: A Survey
• Student Network Learning via Evolutionary Knowledge Distillation
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation