ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies

Abstract

Designing neural architectures requires immense manual efforts. This has promoted the development of neural architecture search (NAS) to automate the design. While previous NAS methods achieve promising results but run slowly, zero-cost proxies run extremely fast but are less promising. Therefore, it is of great potential to accelerate NAS via those zero-cost proxies. The existing method has two limitations, which are unforeseeable reliability and one-shot usage. To address the limitations, we present ProxyBO, an efficient Bayesian optimization (BO) framework that utilizes the zero-cost proxies to accelerate neural architecture search. We apply the generalization ability measurement to estimate the fitness of proxies on the task during each iteration and design a novel acquisition function to combine BO with zero-cost proxies based on their dynamic influence. Extensive empirical studies show that ProxyBO consistently outperforms competitive baselines on five tasks from three public benchmarks. Concretely, ProxyBO achieves up to 5.41x and 3.86x speedups over the state-of-the-art approaches REA and BRP-NAS.

arXiv commentAccepted by AAAI 2023

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Practical Bayesian Optimization of Machine Learning Algorithms
• DARTS: Differentiable Architecture Search
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Neural Architecture Optimization
• Efficient Architecture Search by Network Transformation
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• The Evolved Transformer
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Faster gaze prediction with dense networks and Fisher pruning
• BRP-NAS: Prediction-based NAS using GCNs
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
• MFES-HB: Efficient Hyperband with Multi-Fidelity Quality Measurements