Distilling Knowledge via Knowledge Review
arXiv:2104.09044
Abstract
Knowledge distillation transfers knowledge from the teacher network to the student one, with the goal of greatly improving the performance of the student network. Previous methods mostly focus on proposing feature transformation and loss functions between the same level's features to improve the effectiveness. We differently study the factor of connection path cross levels between teacher and student networks, and reveal its great importance. For the first time in knowledge distillation, cross-stage connection paths are proposed. Our new review mechanism is effective and structurally simple. Our finally designed nested and compact framework requires negligible computation overhead, and outperforms other methods on a variety of tasks. We apply our method to classification, object detection, and instance segmentation tasks. All of them witness significant student network performance improvement. Code is available at https://github.com/Jia-Research-Lab/ReviewKD
CVPR 2021
References in corpus (13)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Focal Loss for Dense Object Detection
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Feature Pyramid Networks for Object Detection
- Pyramid Scene Parsing Network
- Paraphrasing Complex Network: Network Compression via Factor Transfer
- Contrastive Representation Distillation
- Relational Knowledge Distillation
- Variational Information Distillation for Knowledge Transfer