A Comprehensive Overhaul of Feature Distillation

Abstract

We investigate the design aspects of feature distillation methods achieving network compression and propose a novel feature distillation method in which the distillation loss is designed to make a synergy among various aspects: teacher transform, student transform, distillation feature position and distance function. Our proposed distillation loss includes a feature transform with a newly designed margin ReLU, a new distillation feature position, and a partial L2 distance function to skip redundant information giving adverse effects to the compression of student. In ImageNet, our proposed method achieves 21.65% of top-1 error with ResNet50, which outperforms the performance of the teacher network, ResNet152. Our proposed method is evaluated on various tasks such as image classification, object detection and semantic segmentation and achieves a significant performance improvement in all tasks. The code is available at https://sites.google.com/view/byeongho-heo/overhaul

arXiv commentAccepted at ICCV 2019

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• NSML: Meet the MLaaS platform with a real-world case study
• Knowledge Transfer with Jacobian Matching
• Expressiveness of Rectifier Networks

Cited by in corpus (2)

• QKD: Quantization-aware Knowledge Distillation
• Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model