BRP-NAS: Prediction-based NAS using GCNs

Abstract

Neural architecture search (NAS) enables researchers to automatically explore broad design spaces in order to improve efficiency of neural networks. This efficiency is especially important in the case of on-device deployment, where improvements in accuracy should be balanced out with computational demands of a model. In practice, performance metrics of model are computationally expensive to obtain. Previous work uses a proxy (e.g., number of operations) or a layer-wise measurement of neural network layers to estimate end-to-end hardware performance but the imprecise prediction diminishes the quality of NAS. To address this problem, we propose BRP-NAS, an efficient hardware-aware NAS enabled by an accurate performance predictor-based on graph convolutional network (GCN). What is more, we investigate prediction quality on different metrics and show that sample efficiency of the predictor-based NAS can be improved by considering binary relations of models and an iterative data selection strategy. We show that our proposed method outperforms all prior methods on NAS-Bench-101 and NAS-Bench-201, and that our predictor can consistently learn to extract useful features from the DARTS search space, improving upon the second-order baseline. Finally, to raise awareness of the fact that accurate latency estimation is not a trivial task, we release LatBench -- a latency dataset of NAS-Bench-201 models running on a broad range of devices.

arXiv commentPublished at NeurIPS 2020

References in corpus (10)

• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• Efficient Architecture Search by Network Transformation
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Path-Level Network Transformation for Efficient Architecture Search
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Neural Predictor for Neural Architecture Search

Cited by in corpus (11)

• Zero-Cost Proxies for Lightweight NAS
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• One Proxy Device Is Enough for Hardware-Aware Neural Architecture Search
• Stronger NAS with Weaker Predictors
• Learning Versatile Neural Architectures by Propagating Network Codes
• SCGG: A Deep Structure-Conditioned Graph Generative Model
• EH-DNAS: End-to-End Hardware-aware Differentiable Neural Architecture Search
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• FLASH: Fast Neural Architecture Search with Hardware Optimization
• Cascade Bagging for Accuracy Prediction with Few Training Samples