Differential Evolution for Neural Architecture Search

Abstract

Neural architecture search (NAS) methods rely on a search strategy for deciding which architectures to evaluate next and a performance estimation strategy for assessing their performance (e.g., using full evaluations, multi-fidelity evaluations, or the one-shot model). In this paper, we focus on the search strategy. We introduce the simple yet powerful evolutionary algorithm of differential evolution to the NAS community. Using the simplest performance evaluation strategy of full evaluations, we comprehensively compare this search strategy to regularized evolution and Bayesian optimization and demonstrate that it yields improved and more robust results for 13 tabular NAS benchmarks based on NAS-Bench-101, NAS-Bench-1Shot1, NAS-Bench-201 and NAS-HPO bench.

References in corpus (6)

• Neural Architecture Search: A Survey
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization

Cited by in corpus (4)

• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Fisher Task Distance and Its Application in Neural Architecture Search
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor