Bag of Tricks for Neural Architecture Search
arXiv:2107.03719
Abstract
While neural architecture search methods have been successful in previous years and led to new state-of-the-art performance on various problems, they have also been criticized for being unstable, being highly sensitive with respect to their hyperparameters, and often not performing better than random search. To shed some light on this issue, we discuss some practical considerations that help improve the stability, efficiency and overall performance.
References in corpus (16)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Improved Regularization of Convolutional Neural Networks with Cutout
- DARTS: Differentiable Architecture Search
- Searching for Activation Functions
- Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
- Shake-Shake regularization
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Random Search and Reproducibility for Neural Architecture Search
- NAS evaluation is frustratingly hard
- Exploring Randomly Wired Neural Networks for Image Recognition
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
- Rethinking Architecture Selection in Differentiable NAS
- HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark
- Neural Architecture Generator Optimization
- Few-shot Neural Architecture Search