NASGEM: Neural Architecture Search via Graph Embedding Method

Abstract

Neural Architecture Search (NAS) automates and prospers the design of neural networks. Estimator-based NAS has been proposed recently to model the relationship between architectures and their performance to enable scalable and flexible search. However, existing estimator-based methods encode the architecture into a latent space without considering graph similarity. Ignoring graph similarity in node-based search space may induce a large inconsistency between similar graphs and their distance in the continuous encoding space, leading to inaccurate encoding representation and/or reduced representation capacity that can yield sub-optimal search results. To preserve graph correlation information in encoding, we propose NASGEM which stands for Neural Architecture Search via Graph Embedding Method. NASGEM is driven by a novel graph embedding method equipped with similarity measures to capture the graph topology information. By precisely estimating the graph distance and using an auxiliary Weisfeiler-Lehman kernel to guide the encoding, NASGEM can utilize additional structural information to get more accurate graph representation to improve the search efficiency. GEMNet, a set of networks discovered by NASGEM, consistently outperforms networks crafted by existing search methods in classification tasks, i.e., with 0.4%-3.6% higher accuracy while having 11%- 21% fewer Multiply-Accumulates. We further transfer GEMNet for COCO object detection. In both one-stage and twostage detectors, our GEMNet surpasses its manually-crafted and automatically-searched counterparts.

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Graph Embedding Techniques, Applications, and Performance: A Survey
• DARTS: Differentiable Architecture Search
• Neural Architecture Optimization
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Analyzing noise in autoencoders and deep networks
• Discovering Neural Wirings
• Accelerating Neural Architecture Search using Performance Prediction
• Exactly Computing the Local Lipschitz Constant of ReLU Networks
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• Neural Predictor for Neural Architecture Search

Cited by in corpus (5)

• Neural Predictor based Quantum Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• Third ArchEdge Workshop: Exploring the Design Space of Efficient Deep Neural Networks