Student Network Learning via Evolutionary Knowledge Distillation
arXiv:2103.13811
Abstract
Knowledge distillation provides an effective way to transfer knowledge via teacher-student learning, where most existing distillation approaches apply a fixed pre-trained model as teacher to supervise the learning of student network. This manner usually brings in a big capability gap between teacher and student networks during learning. Recent researches have observed that a small teacher-student capability gap can facilitate knowledge transfer. Inspired by that, we propose an evolutionary knowledge distillation approach to improve the transfer effectiveness of teacher knowledge. Instead of a fixed pre-trained teacher, an evolutionary teacher is learned online and consistently transfers intermediate knowledge to supervise student network learning on-the-fly. To enhance intermediate knowledge representation and mimicking, several simple guided modules are introduced between corresponding teacher-student blocks. In this way, the student can simultaneously obtain rich internal knowledge and capture its growth process, leading to effective student network learning. Extensive experiments clearly demonstrate the effectiveness of our approach as well as good adaptability in the low-resolution and few-sample visual recognition scenarios.
References in corpus (25)
- Distilling the Knowledge in a Neural Network
- Wide Residual Networks
- Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
- Do Deep Nets Really Need to be Deep?
- Data Augmentation using Random Image Cropping and Patching for Deep CNNs
- Born Again Neural Networks
- SphereFace: Deep Hypersphere Embedding for Face Recognition
- CosFace: Large Margin Cosine Loss for Deep Face Recognition
- Model compression via distillation and quantization
- Low-resolution Face Recognition in the Wild via Selective Knowledge Distillation
- On Low-Resolution Face Recognition in the Wild: Comparisons and New Techniques
- Large scale distributed neural network training through online distillation
- Recent Advances in Deep Learning: An Overview
- Moonshine: Distilling with Cheap Convolutions
- Feature-map-level Online Adversarial Knowledge Distillation
- Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
- ResKD: Residual-Guided Knowledge Distillation
- MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
- Relational Knowledge Distillation
- Studying Very Low Resolution Recognition Using Deep Networks
- Knowledge Adaptation: Teaching to Adapt
- Knowledge Distillation via Route Constrained Optimization
- Variational Information Distillation for Knowledge Transfer
- Regularizing Class-wise Predictions via Self-knowledge Distillation
- Explaining Knowledge Distillation by Quantifying the Knowledge