Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search
arXiv:2010.15821
Abstract
One-shot weight sharing methods have recently drawn great attention in neural architecture search due to high efficiency and competitive performance. However, weight sharing across models has an inherent deficiency, i.e., insufficient training of subnetworks in hypernetworks. To alleviate this problem, we present a simple yet effective architecture distillation method. The central idea is that subnetworks can learn collaboratively and teach each other throughout the training process, aiming to boost the convergence of individual models. We introduce the concept of prioritized path, which refers to the architecture candidates exhibiting superior performance during training. Distilling knowledge from the prioritized paths is able to boost the training of subnetworks. Since the prioritized paths are changed on the fly depending on their performance and complexity, the final obtained paths are the cream of the crop. We directly select the most promising one from the prioritized paths as the final architecture, without using other complex search methods, such as reinforcement learning or evolution algorithms. The experiments on ImageNet verify such path distillation method can improve the convergence ratio and performance of the hypernetwork, as well as boosting the training of subnetworks. The discovered architectures achieve superior performance compared to the recent MobileNetV3 and EfficientNet families under aligned settings. Moreover, the experiments on object detection and more challenging search space show the generality and robustness of the proposed method. Code and models are available at https://github.com/microsoft/cream.git.
arXiv commentNeurIPS 2020, code url: https://github.com/microsoft/Cream
References in corpus (13)
- Distilling the Knowledge in a Neural Network
- Neural Architecture Search with Reinforcement Learning
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- MixConv: Mixed Depthwise Convolutional Kernels
- A Survey on Neural Architecture Search
- FasterSeg: Searching for Faster Real-time Semantic Segmentation
- BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
- Blockwisely Supervised Neural Architecture Search with Knowledge Distillation
- SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization
- MixPath: A Unified Approach for One-shot Neural Architecture Search
- Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection