Deep Model Compression: Distilling Knowledge from Noisy Teachers

Abstract

The remarkable successes of deep learning models across various applications have resulted in the design of deeper networks that can solve complex problems. However, the increasing depth of such models also results in a higher storage and runtime complexity, which restricts the deployability of such very deep models on mobile and portable devices, which have limited storage and battery capacity. While many methods have been proposed for deep model compression in recent years, almost all of them have focused on reducing storage complexity. In this work, we extend the teacher-student framework for deep model compression, since it has the potential to address runtime and train time complexity too. We propose a simple methodology to include a noise-based regularizer while training the student from the teacher, which provides a healthy improvement in the performance of the student network. Our experiments on the CIFAR-10, SVHN and MNIST datasets show promising improvement, with the best performance on the CIFAR-10 dataset. We also conduct a comprehensive empirical evaluation of the proposed method under related settings on the CIFAR-10 dataset to show the promise of the proposed approach.

9 pages, 3 figures

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• FitNets: Hints for Thin Deep Nets
• Do Deep Nets Really Need to be Deep?
• Compressing Deep Convolutional Networks using Vector Quantization
• Tensorizing Neural Networks
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Lazy Evaluation of Convolutional Filters

Cited by in corpus (35)

• Knowledge Distillation: A Survey
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• Edge Intelligence: Architectures, Challenges, and Applications
• ResKD: Residual-Guided Knowledge Distillation
• Sobolev Training for Neural Networks
• Relational Knowledge Distillation
• Adaptive Neural Network-Based Approximation to Accelerate Eulerian Fluid Simulation
• Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks
• Sub-Band Knowledge Distillation Framework for Speech Enhancement
• Computing Systems for Autonomous Driving: State-of-the-Art and Challenges
• Preparing Lessons: Improve Knowledge Distillation with Better Supervision
• Model Distillation with Knowledge Transfer from Face Classification to Alignment and Verification
• OpenEI: An Open Framework for Edge Intelligence
• Shift-based Primitives for Efficient Convolutional Neural Networks
• Learnable Boundary Guided Adversarial Training
• All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
• MarginDistillation: distillation for margin-based softmax
• Triplet Distillation for Deep Face Recognition
• ADD: Augmented Disentanglement Distillation Framework for Improving Stock Trend Forecasting
• Correlation Congruence for Knowledge Distillation
• Extreme Low Resolution Activity Recognition with Confident Spatial-Temporal Attention Transfer
• Efficient Crowd Counting via Structured Knowledge Transfer
• ESPN: Extremely Sparse Pruned Networks
• KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• The Pupil Has Become the Master: Teacher-Student Model-Based Word Embedding Distillation with Ensemble Learning
• RDPD: Rich Data Helps Poor Data via Imitation
• Light Multi-segment Activation for Model Compression
• Interpretable Few-Shot Learning via Linear Distillation
• Implicit Priors for Knowledge Sharing in Bayesian Neural Networks
• Efficient Fusion of Sparse and Complementary Convolutions
• CHEER: Rich Model Helps Poor Model via Knowledge Infusion
• A Studious Approach to Semi-Supervised Learning
• Learning Fast Matching Models from Weak Annotations