Search for Better Students to Learn Distilled Knowledge

Abstract

Knowledge Distillation, as a model compression technique, has received great attention. The knowledge of a well-performed teacher is distilled to a student with a small architecture. The architecture of the small student is often chosen to be similar to their teacher's, with fewer layers or fewer channels, or both. However, even with the same number of FLOPs or parameters, the students with different architecture can achieve different generalization ability. The configuration of a student architecture requires intensive network architecture engineering. In this work, instead of designing a good student architecture manually, we propose to search for the optimal student automatically. Based on L1-norm optimization, a subgraph from the teacher network topology graph is selected as a student, the goal of which is to minimize the KL-divergence between student's and teacher's outputs. We verify the proposal on CIFAR10 and CIFAR100 datasets. The empirical experiments show that the learned student architecture achieves better performance than ones specified manually. We also visualize and understand the architecture of the found student.

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• DARTS: Differentiable Architecture Search
• Rethinking the Value of Network Pruning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Moonshine: Distilling with Cheap Convolutions
• Exploring Randomly Wired Neural Networks for Image Recognition

Cited by in corpus (6)

• Knowledge Distillation: A Survey
• ResKD: Residual-Guided Knowledge Distillation
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Simple Distillation Baselines for Improving Small Self-supervised Models
• Introspective Learning by Distilling Knowledge from Online Self-explanation
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family