AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family

Abstract

State-of-the-art results in deep learning have been improving steadily, in good part due to the use of larger models. However, widespread use is constrained by device hardware limitations, resulting in a substantial performance gap between state-of-the-art models and those that can be effectively deployed on small devices. While Knowledge Distillation (KD) theoretically enables small student models to emulate larger teacher models, in practice selecting a good student architecture requires considerable human expertise. Neural Architecture Search (NAS) appears as a natural solution to this problem but most approaches can be inefficient, as most of the computation is spent comparing architectures sampled from the same distribution, with negligible differences in performance. In this paper, we propose to instead search for a family of student architectures sharing the property of being good at learning from a given teacher. Our approach AutoKD, powered by Bayesian Optimization, explores a flexible graph-based search space, enabling us to automatically learn the optimal student architecture distribution and KD parameters, while being 20x more sample efficient compared to existing state-of-the-art. We evaluate our method on 3 datasets; on large images specifically, we reach the teacher performance while using 3x less memory and 10x less parameters. Finally, while AutoKD uses the traditional KD loss, it outperforms more advanced KD variants using hand-designed students.

arXiv comment12 pages, 8 figures

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• DARTS: Differentiable Architecture Search
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• Large-Scale Evolution of Image Classifiers
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Model compression via distillation and quantization
• NAS evaluation is frustratingly hard
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• MANAS: Multi-Agent Neural Architecture Search
• Search for Better Students to Learn Distilled Knowledge
• Does Knowledge Distillation Really Work?
• Search to Distill: Pearls are Everywhere but not the Eyes