Zero-Cost Proxies for Lightweight NAS

Abstract

Neural Architecture Search (NAS) is quickly becoming the standard methodology to design neural network models. However, NAS is typically compute-intensive because multiple models need to be evaluated before choosing the best one. To reduce the computational power and time needed, a proxy task is often used for evaluating each model instead of full training. In this paper, we evaluate conventional reduced-training proxies and quantify how well they preserve ranking between multiple models during search when compared with the rankings produced by final trained accuracy. We propose a series of zero-cost proxies, based on recent pruning literature, that use just a single minibatch of training data to compute a model's score. Our zero-cost proxies use 3 orders of magnitude less computation but can match and even outperform conventional proxies. For example, Spearman's rank correlation coefficient between final validation accuracy and our best zero-cost proxy on NAS-Bench-201 is 0.82, compared to 0.61 for EcoNAS (a recently proposed reduced-training proxy). Finally, we use these zero-cost proxies to enhance existing NAS search algorithms such as random search, reinforcement learning, evolutionary search and predictor-based search. For all search methodologies and across three different NAS datasets, we are able to significantly improve sample efficiency, and thereby decrease computation, by using our zero-cost proxies. For example on NAS-Bench-101, we achieved the same accuracy 4$\times$ quicker than the best previous result. Our code is made public at: https://github.com/mohsaied/zero-cost-nas.

ICLR 2021

References in corpus (12)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DARTS: Differentiable Architecture Search
• MixConv: Mixed Depthwise Convolutional Kernels
• Efficient Architecture Search by Network Transformation
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Faster gaze prediction with dense networks and Fisher pruning
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• BRP-NAS: Prediction-based NAS using GCNs
• Neural Architecture Search without Training
• AtomNAS: Fine-Grained End-to-End Neural Architecture Search
• Neural Predictor for Neural Architecture Search
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget

Cited by in corpus (7)

• Machine Learning for Microcontroller-Class Hardware: A Review
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• GENNAPE: Towards Generalized Neural Architecture Performance Estimators
• A General-Purpose Transferable Predictor for Neural Architecture Search
• AIO-P: Expanding Neural Performance Predictors Beyond Image Classification
• A Survey on Green Deep Learning