Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning

Abstract

The improvement in the performance of efficient and lightweight models (i.e., the student model) is achieved through knowledge distillation (KD), which involves transferring knowledge from more complex models (i.e., the teacher model). However, most existing KD techniques rely on Kullback-Leibler (KL) divergence, which has certain limitations. First, if the teacher distribution has high entropy, the KL divergence's mode-averaging nature hinders the transfer of sufficient target information. Second, when the teacher distribution has low entropy, the KL divergence tends to excessively focus on specific modes, which fails to convey an abundant amount of valuable knowledge to the student. Consequently, when dealing with datasets that contain numerous confounding or challenging samples, student models may struggle to acquire sufficient knowledge, resulting in subpar performance. Furthermore, in previous KD approaches, we observed that data augmentation, a technique aimed at enhancing a model's generalization, can have an adverse impact. Therefore, we propose a Robustness-Reinforced Knowledge Distillation (R2KD) that leverages correlation distance and network pruning. This approach enables KD to effectively incorporate data augmentation for performance improvement. Extensive experiments on various datasets, including CIFAR-100, FGVR, TinyImagenet, and ImageNet, demonstrate our method's superiority over current state-of-the-art methods.

13 pages, 7 figures

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Knowledge Distillation: A Survey
• Improved Regularization of Convolutional Neural Networks with Cutout
• FitNets: Hints for Thin Deep Nets
• Effect of Different Distance Measures on the Performance of K-Means Algorithm: An Experimental Study in Matlab
• Knowledge Distillation from A Stronger Teacher
• What Do Compressed Deep Neural Networks Forget?
• PKD: General Distillation Framework for Object Detectors via Pearson Correlation Coefficient
• Self-Damaging Contrastive Learning
• An Empirical Analysis of the Impact of Data Augmentation on Knowledge Distillation
• Maximizing Discrimination Capability of Knowledge Distillation with Energy Function

Cited by in corpus (2)

• Maximizing Discrimination Capability of Knowledge Distillation with Energy Function
• FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer