An Embarrassingly Simple Approach for Knowledge Distillation

Abstract

Knowledge Distillation (KD) aims at improving the performance of a low-capacity student model by inheriting knowledge from a high-capacity teacher model. Previous KD methods typically train a student by minimizing a task-related loss and the KD loss simultaneously, using a pre-defined loss weight to balance these two terms. In this work, we propose to first transfer the backbone knowledge from a teacher to the student, and then only learn the task-head of the student network. Such a decomposition of the training process circumvents the need of choosing an appropriate loss weight, which is often difficult in practice, and thus makes it easier to apply to different datasets and tasks. Importantly, the decomposition permits the core of our method, Stage-by-Stage Knowledge Distillation (SSKD), which facilitates progressive feature mimicking from teacher to student. Extensive experiments on CIFAR-100 and ImageNet suggest that SSKD significantly narrows down the performance gap between student and teacher, outperforming state-of-the-art approaches. We also demonstrate the generalization ability of SSKD on other challenging benchmarks, including face recognition on IJB-A dataset as well as object detection on COCO dataset.

arXiv comment8 pages and 5 figures

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Learning Face Representation from Scratch
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• Do Deep Nets Really Need to be Deep?
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Knowledge Distillation by On-the-Fly Native Ensemble
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning

Cited by in corpus (8)

• Knowledge Distillation in Deep Learning and its Applications
• Autoencoders on FPGAs for real-time, unsupervised new physics detection at 40 MHz at the Large Hadron Collider
• Residual Knowledge Distillation
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Inter-Region Affinity Distillation for Road Marking Segmentation
• Prime-Aware Adaptive Distillation
• Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression
• Knowledge Distillation By Sparse Representation Matching