Trainless Model Performance Estimation for Neural Architecture Search

Abstract

Neural architecture search has become an indispensable part of the deep learning field. Modern methods allow to find one of the best performing architectures, or to build one from scratch, but they typically make decisions based on the trained accuracy information. In the present article we explore instead how the architectural component of a neural network affects its prediction power. We focus on relationships between the trained accuracy of an architecture and its accuracy prior to training, by considering statistics over multiple initialisations. We observe that minimising the coefficient of variation of the untrained accuracy, $C{V}_U$, consistently leads to better performing architectures. We test the $C{V}_U$ as a neural architecture search scoring metric using the NAS-Bench-201 database of trained neural architectures. The architectures with the lowest $C{V}_U$ value have on average an accuracy of $91.90\pm 2.27$, $64.08\pm 5.63$ and $38.76\pm 6.62$ for CIFAR-10, CIFAR-100 and a downscaled version of ImageNet, respectively. Since these values are statistically above the random baseline, we make a conclusion that a good architecture should be stable against weights initialisations. It takes about $190$ s for CIFAR-10 and CIFAR-100 and $133.9$ s for ImageNet16-120 to process $100$ architectures, on a batch of $256$ images, with $100$ initialisations.

References in corpus (13)

• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• A Tutorial on Bayesian Optimization
• Hierarchical Representations for Efficient Architecture Search
• BOHB: Robust and Efficient Hyperparameter Optimization at Scale
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• Peephole: Predicting Network Performance Before Training
• Efficient Multi-objective Neural Architecture Search via Lamarckian Evolution
• Big Neural Networks Waste Capacity
• Neural Architecture Search without Training

Cited by in corpus (1)

• Neural Architecture Search: Two Constant Shared Weights Initialisations