Residual Knowledge Distillation

Abstract

Knowledge distillation (KD) is one of the most potent ways for model compression. The key idea is to transfer the knowledge from a deep teacher model (T) to a shallower student (S). However, existing methods suffer from performance degradation due to the substantial gap between the learning capacities of S and T. To remedy this problem, this work proposes Residual Knowledge Distillation (RKD), which further distills the knowledge by introducing an assistant (A). Specifically, S is trained to mimic the feature maps of T, and A aids this process by learning the residual error between them. In this way, S and A complement with each other to get better knowledge from T. Furthermore, we devise an effective method to derive S and A from a given model without increasing the total computational cost. Extensive experiments show that our approach achieves appealing results on popular classification datasets, CIFAR-100 and ImageNet, surpassing state-of-the-art methods.

9 pages, 3 figures, 3 tables

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• An Embarrassingly Simple Approach for Knowledge Distillation

Cited by in corpus (5)

• Knowledge Distillation: A Survey
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation
• Privileged Knowledge Distillation for Online Action Detection
• Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation