Knowledge distillation via adaptive instance normalization
arXiv:2003.04289
Abstract
This paper addresses the problem of model compression via knowledge distillation. To this end, we propose a new knowledge distillation method based on transferring feature statistics, specifically the channel-wise mean and variance, from the teacher to the student. Our method goes beyond the standard way of enforcing the mean and variance of the student to be similar to those of the teacher through an loss, which we found it to be of limited effectiveness. Specifically, we propose a new loss based on adaptive instance normalization to effectively transfer the feature statistics. The main idea is to transfer the learned statistics back to the teacher via adaptive instance normalization (conditioned on the student) and let the teacher network "evaluate" via a loss whether the statistics learned by the student are reliably transferred. We show that our distillation method outperforms other state-of-the-art distillation methods over a large set of experimental settings including different (a) network architectures, (b) teacher-student capacities, (c) datasets, and (d) domains.
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Neural Architecture Search with Reinforcement Learning
- FitNets: Hints for Thin Deep Nets
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- DARTS: Differentiable Architecture Search
- Revisiting Batch Normalization For Practical Domain Adaptation
- Generative Moment Matching Networks
- Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
- On the Power of Over-parametrization in Neural Networks with Quadratic Activation
- XNOR-Net++: Improved Binary Neural Networks
- Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net