Cross-Layer Distillation with Semantic Calibration
arXiv:2012.03236
Abstract
Knowledge distillation is a technique to enhance the generalization ability of a student model by exploiting outputs from a teacher model. Recently, feature-map based variants explore knowledge transfer between manually assigned teacher-student pairs in intermediate layers for further improvement. However, layer semantics may vary in different neural networks and semantic mismatch in manual layer associations will lead to performance degeneration due to negative regularization. To address this issue, we propose Semantic Calibration for cross-layer Knowledge Distillation (SemCKD), which automatically assigns proper target layers of the teacher model for each student layer with an attention mechanism. With a learned attention distribution, each student layer distills knowledge contained in multiple teacher layers rather than a specific intermediate layer for appropriate cross-layer supervision. We further provide theoretical analysis of the association weights and conduct extensive experiments to demonstrate the effectiveness of our approach. Code is avaliable at \url{https://github.com/DefangChen/SemCKD}.
arXiv commentExtended version of our previous AAAI-2021 paper
References in corpus (9)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Distilling the Knowledge in a Neural Network
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- Compressing Deep Convolutional Networks using Vector Quantization
- Tensorizing Neural Networks
- Similarity of Neural Network Representations Revisited
- Learning What and Where to Transfer