Graph HyperNetworks for Neural Architecture Search

Abstract

Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -- they can search nearly 10 times faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.

arXiv commentICLR 2019

References in corpus (9)

• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• MONAS: Multi-Objective Neural Architecture Search using Reinforcement Learning
• Peephole: Predicting Network Performance Before Training
• Graph Partition Neural Networks for Semi-Supervised Classification

Cited by in corpus (27)

• AutoML: A Survey of the State-of-the-Art
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Review: Deep Learning in Electron Microscopy
• Hypernetwork functional image representation
• Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
• 3D-ANAS: 3D Asymmetric Neural Architecture Search for Fast Hyperspectral Image Classification
• Principled Weight Initialization for Hypernetworks
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• Regularization-Agnostic Compressed Sensing MRI Reconstruction with Hypernetworks
• An Introduction to Neural Architecture Search for Convolutional Networks
• NAS-Navigator: Visual Steering for Explainable One-Shot Deep Neural Network Synthesis
• SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures
• Few-shot Video-to-Video Synthesis
• HyperInverter: Improving StyleGAN Inversion via Hypernetwork
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• ESAI: Efficient Split Artificial Intelligence via Early Exiting Using Neural Architecture Search
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Learning Architectures for Binary Networks
• Parameter Prediction for Unseen Deep Architectures
• Large Scale Neural Architecture Search with Polyharmonic Splines
• Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising
• Evolutionary Algorithm Enhanced Neural Architecture Search for Text-Independent Speaker Verification
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Fast Unsupervised Deep Outlier Model Selection with Hypernetworks