Distilling Knowledge by Mimicking Features

Abstract

Knowledge distillation (KD) is a popular method to train efficient networks ("student") with the help of high-capacity networks ("teacher"). Traditional methods use the teacher's soft logits as extra supervision to train the student network. In this paper, we argue that it is more advantageous to make the student mimic the teacher's features in the penultimate layer. Not only the student can directly learn more effective information from the teacher feature, feature mimicking can also be applied for teachers trained without a softmax layer. Experiments show that it can achieve higher accuracy than traditional KD. To further facilitate feature mimicking, we decompose a feature vector into the magnitude and the direction. We argue that the teacher should give more freedom to the student feature's magnitude, and let the student pay more attention on mimicking the feature direction. To meet this requirement, we propose a loss term based on locality-sensitive hashing (LSH). With the help of this new loss, our method indeed mimics feature directions more accurately, relaxes constraints on feature magnitudes, and achieves state-of-the-art distillation accuracy. We provide theoretical analyses of how LSH facilitates feature direction mimicking, and further extend feature mimicking to multi-label recognition and object detection.

arXiv commentTo appear in IEEE Trans. PAMI

References in corpus (8)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Improved Baselines with Momentum Contrastive Learning
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Unsupervised Representation Learning by Predicting Image Rotations
• NormFace: L2 Hypersphere Embedding for Face Verification
• Knowledge Distillation by On-the-Fly Native Ensemble
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision

Cited by in corpus (5)

• Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• Rethinking Knowledge Distillation in Collaborative Machine Learning: Memory, Knowledge, and Their Interactions
• Towards Real-Time Neural Video Codec for Cross-Platform Application Using Calibration Information
• Ground Reaction Force Estimation via Time-aware Knowledge Distillation