ResKD: Residual-Guided Knowledge Distillation

Abstract

Knowledge distillation, aimed at transferring the knowledge from a heavy teacher network to a lightweight student network, has emerged as a promising technique for compressing neural networks. However, due to the capacity gap between the heavy teacher and the lightweight student, there still exists a significant performance gap between them. In this paper, we see knowledge distillation in a fresh light, using the knowledge gap, or the residual, between a teacher and a student as guidance to train a much more lightweight student, called a res-student. We combine the student and the res-student into a new student, where the res-student rectifies the errors of the former student. Such a residual-guided process can be repeated until the user strikes the balance between accuracy and cost. At inference time, we propose a sample-adaptive strategy to decide which res-students are not necessary for each sample, which can save computational cost. Experimental results show that we achieve competitive performance with 18.04$\%$, 23.14$\%$, 53.59$\%$, and 56.86$\%$ of the teachers' computational costs on the CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets. Finally, we do thorough theoretical and empirical analysis for our method.

The first two authors (Xuewei Li and Songyuan Li) contribute equally. Accepted to IEEE TRANSACTIONS ON IMAGE PROCESSING (TIP)

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• Knowledge Distillation: A Survey
• Popular Ensemble Methods: An Empirical Study
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Deep Model Compression: Distilling Knowledge from Noisy Teachers
• Robust Face Alignment by Multi-order High-precision Hourglass Network
• Fixing the train-test resolution discrepancy: FixEfficientNet
• Search for Better Students to Learn Distilled Knowledge
• Improving Neural Architecture Search Image Classifiers via Ensemble Learning
• AdaSample: Adaptive Sampling of Hard Positives for Descriptor Learning

Cited by in corpus (4)

• Self-Distribution Binary Neural Networks
• Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better
• Student Network Learning via Evolutionary Knowledge Distillation
• Dual Discriminator Adversarial Distillation for Data-free Model Compression