Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
arXiv:2102.01063
Abstract
Accuracy predictor is a key component in Neural Architecture Search (NAS) for ranking architectures. Building a high-quality accuracy predictor usually costs enormous computation. To address this issue, instead of using an accuracy predictor, we propose a novel zero-shot index dubbed Zen-Score to rank the architectures. The Zen-Score represents the network expressivity and positively correlates with the model accuracy. The calculation of Zen-Score only takes a few forward inferences through a randomly initialized network, without training network parameters. Built upon the Zen-Score, we further propose a new NAS algorithm, termed as Zen-NAS, by maximizing the Zen-Score of the target network under given inference budgets. Within less than half GPU day, Zen-NAS is able to directly search high performance architectures in a data-free style. Comparing with previous NAS methods, the proposed Zen-NAS is magnitude times faster on multiple server-side and mobile-side GPU platforms with state-of-the-art accuracy on ImageNet. Our source code and pre-trained models are released on https://github.com/idstcv/ZenNAS.
arXiv commentaccepted by ICCV 2021
References in corpus (22)
- Deep Learning using Rectified Linear Units (ReLU)
- DARTS: Differentiable Architecture Search
- On the Number of Linear Regions of Deep Neural Networks
- AutoAugment: Learning Augmentation Policies from Data
- Large-Scale Evolution of Image Classifiers
- Neural Architecture Optimization
- Designing Neural Network Architectures using Reinforcement Learning
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- Exponential expressivity in deep neural networks through transient chaos
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
- Why Deep Neural Networks for Function Approximation?
- Picking Winning Tickets Before Training by Preserving Gradient Flow
- The power of deeper networks for expressing natural functions
- Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
- Exploring Randomly Wired Neural Networks for Image Recognition
- Complexity of Linear Regions in Deep Networks
- Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
- Semi-Supervised Neural Architecture Search
- Empirical Studies on the Properties of Linear Regions in Deep Neural Networks
- On the Number of Linear Regions of Convolutional Neural Networks
- Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild