Knowledge Distillation via Route Constrained Optimization

Abstract

Distillation-based learning boosts the performance of the miniaturized neural network based on the hypothesis that the representation of a teacher model can be used as structured and relatively weak supervision, and thus would be easily learned by a miniaturized model. However, we find that the representation of a converged heavy model is still a strong constraint for training a small student model, which leads to a high lower bound of congruence loss. In this work, inspired by curriculum learning we consider the knowledge distillation from the perspective of curriculum learning by routing. Instead of supervising the student model with a converged teacher model, we supervised it with some anchor points selected from the route in parameter space that the teacher model passed by, as we called route constrained optimization (RCO). We experimentally demonstrate this simple operation greatly reduces the lower bound of congruence loss for knowledge distillation, hint and mimicking learning. On close-set classification tasks like CIFAR100 and ImageNet, RCO improves knowledge distillation by 2.14% and 1.5% respectively. For the sake of evaluating the generalization, we also test RCO on the open-set face recognition task MegaFace.

References in corpus (13)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• FitNets: Hints for Thin Deep Nets
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Pruning Filters for Efficient ConvNets
• Visualizing the Loss Landscape of Neural Nets
• Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
• Deep Mutual Learning
• Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net
• The Devil of Face Recognition is in the Noise

Cited by in corpus (10)

• Knowledge Distillation: A Survey
• Learning Student-Friendly Teacher Networks for Knowledge Distillation
• Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model
• Search to Distill: Pearls are Everywhere but not the Eyes
• Student Network Learning via Evolutionary Knowledge Distillation
• Real-time Policy Distillation in Deep Reinforcement Learning
• Fixing the Teacher-Student Knowledge Discrepancy in Distillation
• Extending Label Smoothing Regularization with Self-Knowledge Distillation
• KD-Lib: A PyTorch library for Knowledge Distillation, Pruning and Quantization
• Neighbourhood Distillation: On the benefits of non end-to-end distillation