ResKD: Residual-Guided Knowledge Distillation

Abstract

Knowledge distillation, aimed at transferring the knowledge from a heavy teacher network to a lightweight student network, has emerged as a promising technique for compressing neural networks. However, due to the capacity gap between the heavy teacher and the lightweight student, there still exists a significant performance gap between them. In this paper, we see knowledge distillation in a fresh light, using the knowledge gap, or the residual, between a teacher and a student as guidance to train a much more lightweight student, called a res-student. We combine the student and the res-student into a new student, where the res-student rectifies the errors of the former student. Such a residual-guided process can be repeated until the user strikes the balance between accuracy and cost. At inference time, we propose a sample-adaptive strategy to decide which res-students are not necessary for each sample, which can save computational cost. Experimental results show that we achieve competitive performance with 18.04$\%$, 23.14$\%$, 53.59$\%$, and 56.86$\%$ of the teachers' computational costs on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Finally, we do thorough theoretical and empirical analysis for our method.

arXiv commentThe first two authors (Xuewei Li and Songyuan Li) contribute equally. Accepted to IEEE TRANSACTIONS ON IMAGE PROCESSING (TIP)

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Knowledge Distillation: A Survey
• Popular Ensemble Methods: An Empirical Study
• Do Deep Nets Really Need to be Deep?
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Knowledge Distillation by On-the-Fly Native Ensemble
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Self-training with Noisy Student improves ImageNet classification
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Moonshine: Distilling with Cheap Convolutions
• Paraphrasing Complex Network: Network Compression via Factor Transfer
• Robust Face Alignment by Multi-order High-precision Hourglass Network
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Contrastive Representation Distillation
• Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency
• Search for Better Students to Learn Distilled Knowledge
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• Hierarchical Neural Architecture Search via Operator Clustering
• AdaSample: Adaptive Sampling of Hard Positives for Descriptor Learning
• Deep Net Triage: Analyzing the Importance of Network Layers via Structural Compression

Cited by in corpus (4)

• Self-Distribution Binary Neural Networks
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• Student Network Learning via Evolutionary Knowledge Distillation
• Dual Discriminator Adversarial Distillation for Data-free Model Compression