Neighborhood-Aware Neural Architecture Search
arXiv:2105.06369
Abstract
Existing neural architecture search (NAS) methods often return an architecture with good search performance but generalizes poorly to the test setting. To achieve better generalization, we propose a novel neighborhood-aware NAS formulation to identify flat-minima architectures in the search space, with the assumption that flat minima generalize better than sharp minima. The phrase ``flat-minima architecture'' refers to architectures whose performance is stable under small perturbations in the architecture (e.g., replacing a convolution with a skip connection). Our formulation takes the ``flatness'' of an architecture into account by aggregating the performance over the neighborhood of this architecture. We demonstrate a principled way to apply our formulation to existing search algorithms, including sampling-based algorithms and gradient-based algorithms. To facilitate the application to gradient-based algorithms, we also propose a differentiable representation for the neighborhood of architectures. Based on our formulation, we propose neighborhood-aware random search (NA-RS) and neighborhood-aware differentiable architecture search (NA-DARTS). Notably, by simply augmenting DARTS with our formulation, NA-DARTS outperforms DARTS and achieves state-of-the-art performance on established benchmarks, including CIFAR-10, CIFAR-100 and ImageNet.
arXiv commentBMVC 2021
References in corpus (14)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Architecture Search with Reinforcement Learning
- Improved Regularization of Convolutional Neural Networks with Cutout
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- Designing Neural Network Architectures using Reinforcement Learning
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- Understanding and Robustifying Differentiable Architecture Search
- DARTS+: Improved Differentiable Architecture Search with Early Stopping
- Sharp Minima Can Generalize For Deep Nets
- Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
- NAS evaluation is frustratingly hard
- Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
- Learnable Embedding Space for Efficient Neural Architecture Compression