Knowledge Distillation in Deep Learning and its Applications
arXiv:2007.09029 · doi:10.7717/peerj-cs.474
Abstract
Deep learning based models are relatively large, and it is hard to deploy such models on resource-limited devices such as mobile phones and embedded devices. One possible solution is knowledge distillation whereby a smaller model (student model) is trained by utilizing the information from a larger model (teacher model). In this paper, we present a survey of knowledge distillation techniques applied to deep learning models. To compare the performances of different techniques, we propose a new metric called distillation metric. Distillation metric compares different knowledge distillation algorithms based on sizes and accuracy scores. Based on the survey, some interesting conclusions are drawn and presented in this paper.
arXiv comment9 pages
References in corpus (22)
- Distilling the Knowledge in a Neural Network
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Wide Residual Networks
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Learning Face Representation from Scratch
- A Survey of Model Compression and Acceleration for Deep Neural Networks
- Born Again Neural Networks
- Knowledge Distillation by On-the-Fly Native Ensemble
- Model compression via distillation and quantization
- NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
- Data-Free Knowledge Distillation for Deep Neural Networks
- Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
- What Do Compressed Deep Neural Networks Forget?
- Zero-Shot Knowledge Distillation in Deep Networks
- Feature-map-level Online Adversarial Knowledge Distillation
- Few-shot learning of neural networks from scratch by pseudo example optimization
- Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net
- Expressiveness of Rectifier Networks
- Feature Fusion for Online Mutual Knowledge Distillation
- Learning Lightweight Pedestrian Detector with Hierarchical Knowledge Distillation
- An Embarrassingly Simple Approach for Knowledge Distillation