Spirit Distillation: A Model Compression Method with Multi-domain Knowledge Transfer

Abstract

Recent applications pose requirements of both cross-domain knowledge transfer and model compression to machine learning models due to insufficient training data and limited computational resources. In this paper, we propose a new knowledge distillation model, named Spirit Distillation (SD), which is a model compression method with multi-domain knowledge transfer. The compact student network mimics out a representation equivalent to the front part of the teacher network, through which the general knowledge can be transferred from the source domain (teacher) to the target domain (student). To further improve the robustness of the student, we extend SD to Enhanced Spirit Distillation (ESD) in exploiting a more comprehensive knowledge by introducing the proximity domain which is similar to the target domain for feature extraction. Results demonstrate that our method can boost mIOU and high-precision accuracy by 1.4% and 8.2% respectively with 78.2% segmentation variance, and can gain a precise compact network with only 41.8% FLOPs.

arXiv comment13 pages, 4 figures, 4 tables. arXiv admin note: text overlap with arXiv:2103.13733

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• How transferable are features in deep neural networks?
• FitNets: Hints for Thin Deep Nets
• Meta-Learning with Latent Embedding Optimization
• Contrastive Representation Distillation
• Improving Fast Segmentation With Teacher-student Learning