Knowledge Distillation By Sparse Representation Matching

Abstract

Knowledge Distillation refers to a class of methods that transfers the knowledge from a teacher network to a student network. In this paper, we propose Sparse Representation Matching (SRM), a method to transfer intermediate knowledge obtained from one Convolutional Neural Network (CNN) to another by utilizing sparse representation learning. SRM first extracts sparse representations of the hidden features of the teacher CNN, which are then used to generate both pixel-level and image-level labels for training intermediate feature maps of the student network. We formulate SRM as a neural processing block, which can be efficiently optimized using stochastic gradient descent and integrated into any CNN in a plug-and-play manner. Our experiments demonstrate that SRM is robust to architectural differences between the teacher and student networks, and outperforms other KD techniques across several datasets.

arXiv comment9 pages

References in corpus (24)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• YOLOv3: An Incremental Improvement
• Neural Architecture Search with Reinforcement Learning
• Striving for Simplicity: The All Convolutional Net
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• A survey of sparse representation: algorithms and applications
• Task-Driven Dictionary Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• DeepLOB: Deep Convolutional Neural Networks for Limit Order Books
• Temporal Attention augmented Bilinear Network for Financial Time-Series Data Analysis
• Data-Free Knowledge Distillation for Deep Neural Networks
• Heterogeneous Multilayer Generalized Operational Perceptron
• Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
• Automated Pruning for Deep Neural Network Compression
• Few-shot learning of neural networks from scratch by pseudo example optimization
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Multilinear Compressive Learning
• An Embarrassingly Simple Approach for Knowledge Distillation
• Multilinear Compressive Learning with Prior Knowledge