AtomNAS: Fine-Grained End-to-End Neural Architecture Search

Abstract

Search space design is very critical to neural architecture search (NAS) algorithms. We propose a fine-grained search space comprised of atomic blocks, a minimal search unit that is much smaller than the ones used in recent NAS algorithms. This search space allows a mix of operations by composing different types of atomic blocks, while the search space in previous methods only allows homogeneous operations. Based on this search space, we propose a resource-aware architecture search framework which automatically assigns the computational resources (e.g., output channel numbers) for each operation by jointly considering the performance and the computational cost. In addition, to accelerate the search process, we propose a dynamic network shrinkage technique which prunes the atomic blocks with negligible influence on outputs on the fly. Instead of a search-and-retrain two-stage paradigm, our method simultaneously searches and trains the target architecture. Our method achieves state-of-the-art performance under several FLOPs configurations on ImageNet with a small searching cost. We open our entire codebase at: https://github.com/meijieru/AtomNAS.

arXiv commentICLR 2020 camera ready version

References in corpus (7)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DARTS: Differentiable Architecture Search
• Rethinking the Value of Network Pruning
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• AutoAugment: Learning Augmentation Policies from Data
• MixConv: Mixed Depthwise Convolutional Kernels
• Single Path One-Shot Neural Architecture Search with Uniform Sampling

Cited by in corpus (19)

• AutoML: A Survey of the State-of-the-Art
• Neural Architecture Transfer
• Zero-Cost Proxies for Lightweight NAS
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• Few-shot Neural Architecture Search
• Learning Versatile Neural Architectures by Propagating Network Codes
• DrNAS: Dirichlet Neural Architecture Search
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• NetAdaptV2: Efficient Neural Architecture Search with Fast Super-Network Training and Architecture Optimization
• Rethinking Neural Operations for Diverse Tasks
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• Neural Architecture Search via Combinatorial Multi-Armed Bandit
• Differentiable Neural Architecture Search with Morphism-based Transformable Backbone Architectures