A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models

Abstract

This paper aims to provide a selective survey about knowledge distillation(KD) framework for researchers and practitioners to take advantage of it for developing new optimized models in the deep neural network field. To this end, we give a brief overview of knowledge distillation and some related works including learning using privileged information(LUPI) and generalized distillation(GD). Even though knowledge distillation based on the teacher-student architecture was initially devised as a model compression technique, it has found versatile applications over various frameworks. In this paper, we review the characteristics of knowledge distillation from the hypothesis that the three important ingredients of knowledge distillation are distilled knowledge and loss,teacher-student paradigm, and the distillation process. In addition, we survey the versatility of the knowledge distillation by studying its direct applications and its usage in combination with other deep learning paradigms. Finally we present some future works in knowledge distillation including explainable knowledge distillation where the analytical analysis of the performance gain is studied and the self-supervised learning which is a hot research topic in deep learning community.

15 pages, 19 figures

References in corpus (22)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Do Deep Nets Really Need to be Deep?
• Focal Loss for Dense Object Detection
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Object Detectors Emerge in Deep Scene CNNs
• Big Self-Supervised Models are Strong Semi-Supervised Learners
• Learning Using Privileged Information: SVM+ and Weighted SVM
• TinyBERT: Distilling BERT for Natural Language Understanding
• Towards Understanding Knowledge Distillation
• Cascade R-CNN: High Quality Object Detection and Instance Segmentation
• Towards the Limit of Network Quantization
• Learning to Transfer Privileged Information
• Dynamic Capacity Networks
• FrostNet: Towards Quantization-Aware Network Architecture Search
• BABO: Background Activation Black-Out for Efficient Object Detection