GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor

Abstract

In practice, the problems encountered in Neural Architecture Search (NAS) training are not simple problems, but often a series of difficult combinations (wrong compensation estimation, curse of dimension, overfitting, high complexity, etc.). In this paper, we propose a framework to decouple network structure from operator search space, and use two BOHBs to search alternatively. Considering that activation function and initialization are also important parts of neural network, the generalization ability of the model will be affected. We introduce an activation function and an initialization method domain, and add them into the operator search space to form a generalized search space, so as to improve the generalization ability of the child model. We then trained a GCN-based predictor using feedback from the child model. This can not only improve the search efficiency, but also solve the problem of dimension curse. Next, unlike other NAS studies, we used predictors to analyze the stability of different network structures. Finally, we applied our framework to neural structure search and achieved significant improvements on multiple datasets.

arXiv comment12 pages,15 figures

References in corpus (21)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Convolutional Sequence to Sequence Learning
• Efficient Neural Architecture Search via Parameter Sharing
• Designing Neural Network Architectures using Reinforcement Learning
• Shake-Shake regularization
• SNAS: Stochastic Neural Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Hierarchical Representations for Efficient Architecture Search
• Convolutional Neural Fabrics
• Path-Level Network Transformation for Efficient Architecture Search
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Best Practices for Scientific Research on Neural Architecture Search
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Peephole: Predicting Network Performance Before Training
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• APQ: Joint Search for Network Architecture, Pruning and Quantization Policy
• Differential Evolution for Neural Architecture Search
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS