HourNAS: Extremely Fast Neural Architecture Search Through an Hourglass Lens

Abstract

Neural Architecture Search (NAS) refers to automatically design the architecture. We propose an hourglass-inspired approach (HourNAS) for this problem that is motivated by the fact that the effects of the architecture often proceed from the vital few blocks. Acting like the narrow neck of an hourglass, vital blocks in the guaranteed path from the input to the output of a deep neural network restrict the information flow and influence the network accuracy. The other blocks occupy the major volume of the network and determine the overall network complexity, corresponding to the bulbs of an hourglass. To achieve an extremely fast NAS while preserving the high accuracy, we propose to identify the vital blocks and make them the priority in the architecture search. The search space of those non-vital blocks is further shrunk to only cover the candidates that are affordable under the computational resource constraints. Experimental results on the ImageNet show that only using 3 hours (0.1 days) with one GPU, our HourNAS can search an architecture that achieves a 77.0% Top-1 accuracy, which outperforms the state-of-the-art methods.

References in corpus (16)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• DARTS: Differentiable Architecture Search
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Searching for Activation Functions
• Efficient Neural Architecture Search via Parameter Sharing
• MixConv: Mixed Depthwise Convolutional Kernels
• Deep Networks with Stochastic Depth
• Understanding and Robustifying Differentiable Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• Global Sparse Momentum SGD for Pruning Very Deep Neural Networks
• NAS evaluation is frustratingly hard
• Are All Layers Created Equal?
• Meta Architecture Search

Cited by in corpus (5)

• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Stronger NAS with Weaker Predictors
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Discovering Multi-Hardware Mobile Models via Architecture Search