Neural Architecture Optimization

Abstract

Automatic neural architecture design has shown its potential in discovering powerful neural network architectures. Existing methods, no matter based on reinforcement learning or evolutionary algorithms (EA), conduct architecture search in a discrete space, which is highly inefficient. In this paper, we propose a simple and efficient method to automatic neural architecture design based on continuous optimization. We call this new approach neural architecture optimization (NAO). There are three key components in our proposed approach: (1) An encoder embeds/maps neural network architectures into a continuous space. (2) A predictor takes the continuous representation of a network as input and predicts its accuracy. (3) A decoder maps a continuous representation of a network back to its architecture. The performance predictor and the encoder enable us to perform gradient based optimization in the continuous space to find the embedding of a new architecture with potentially better accuracy. Such a better embedding is then decoded to a network by the decoder. Experiments show that the architecture discovered by our method is very competitive for image classification task on CIFAR-10 and language modeling task on PTB, outperforming or on par with the best results of previous architecture search methods with a significantly reduction of computational resources. Specifically we obtain 1.93% test set error rate for CIFAR-10 image classification task and 56.0 test set perplexity of PTB language modeling task. Furthermore, combined with the recent proposed weight sharing mechanism, we discover powerful architecture on CIFAR-10 (with error rate 2.93%) and on PTB (with test set perplexity 56.6), with very limited computational resources (less than 10 GPU hours) for both tasks.

arXiv commentNeurIPS 2018. Code available at: https://github.com/renqianluo/NAO

Cited by in corpus (159)

• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• AutoML: A Survey of the State-of-the-Art
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• A Survey on Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Evaluating the Search Phase of Neural Architecture Search
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Graph HyperNetworks for Neural Architecture Search
• NAS evaluation is frustratingly hard
• BRP-NAS: Prediction-based NAS using GCNs
• Population-Based Black-Box Optimization for Biological Sequence Design
• Neural Predictor based Quantum Architecture Search
• Any equation is a forest: Symbolic genetic algorithm for discovering open-form partial differential equations (SGA-PDE)
• D-VAE: A Variational Autoencoder for Directed Acyclic Graphs
• Auto-Keras: An Efficient Neural Architecture Search System
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• Exploring Randomly Wired Neural Networks for Image Recognition
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Weighted Random Search for CNN Hyperparameter Optimization
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• Probabilistic Neural Architecture Search
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• ASAP: Architecture Search, Anneal and Prune
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• Multi-scale Convolutional Neural Networks for Inverse Problems
• Evolving Normalization-Activation Layers
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Toward Extremely Lightweight Distracted Driver Recognition With Distillation-Based Neural Architecture Search and Knowledge Transfer
• Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data
• Semi-Supervised Neural Architecture Search
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Neural Architecture Search without Training
• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Rethinking Architecture Selection in Differentiable NAS
• Evaluating Efficient Performance Estimators of Neural Architectures
• GOLD-NAS: Gradual, One-Level, Differentiable
• Improving Auto-Augment via Augmentation-Wise Weight Sharing
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Constrained Design of Deep Iris Networks
• Latency-Aware Differentiable Neural Architecture Search
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• XNAS: Neural Architecture Search with Expert Advice
• Generative Multi-Form Bayesian Optimization
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• Neural Predictor for Neural Architecture Search
• Memory-efficient Embedding for Recommendations
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• How Powerful are Performance Predictors in Neural Architecture Search?
• Neural Architecture Generator Optimization
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• One-Shot Neural Architecture Search via Compressive Sensing
• NSGANetV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search
• MvFS: Multi-view Feature Selection for Recommender System
• MixPath: A Unified Approach for One-shot Neural Architecture Search
• AutoPose: Searching Multi-Scale Branch Aggregation for Pose Estimation
• AM-LFS: AutoML for Loss Function Search
• BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget
• Dynamic Embedding Size Search with Minimum Regret for Streaming Recommender System
• Stronger NAS with Weaker Predictors
• To Share or Not To Share: A Comprehensive Appraisal of Weight-Sharing
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• SGAS: Sequential Greedy Architecture Search
• Analyzing and Mitigating Interference in Neural Architecture Search
• A Semi-Supervised Assessor of Neural Architectures
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• Efficient Neural Architecture Search via Proximal Iterations
• Cyclic Differentiable Architecture Search
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• A Variational-Sequential Graph Autoencoder for Neural Architecture Performance Prediction
• NASGEM: Neural Architecture Search via Graph Embedding Method
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• Neural Architecture Optimization with Graph VAE
• Efficient Forward Architecture Search
• Balanced One-shot Neural Architecture Optimization
• Understanding Architectures Learnt by Cell-based Neural Architecture Search
• Techniques for Automated Machine Learning
• Growing Efficient Deep Networks by Structured Continuous Sparsification
• Online Hyper-parameter Learning for Auto-Augmentation Strategy
• An Introduction to Neural Architecture Search for Convolutional Networks
• Few-shot Neural Architecture Search
• Deep unfolding of the weighted MMSE beamforming algorithm
• NASI: Label- and Data-agnostic Neural Architecture Search at Initialization
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Towards Improving the Consistency, Efficiency, and Flexibility of Differentiable Neural Architecture Search
• AutoSF: Searching Scoring Functions for Knowledge Graph Embedding
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• Leveraging End-to-End Speech Recognition with Neural Architecture Search
• Fast Neural Architecture Construction using EnvelopeNets
• A Study of the Learning Progress in Neural Architecture Search Techniques
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• The Differentiable Cross-Entropy Method
• Neural Architecture Search by Estimation of Network Structure Distributions
• HM-NAS: Efficient Neural Architecture Search via Hierarchical Masking
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Rethinking the Number of Channels for the Convolutional Neural Network
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
• Towards Oracle Knowledge Distillation with Neural Architecture Search
• Learning Architectures for Binary Networks
• Learning to Rank Learning Curves
• Light-Weighted CNN for Text Classification
• Contrastive Neural Architecture Search with Neural Architecture Comparators
• Discovering Robust Convolutional Architecture at Targeted Capacity: A Multi-Shot Approach
• FNAS: Uncertainty-Aware Fast Neural Architecture Search
• Achieving Real-Time LiDAR 3D Object Detection on a Mobile Device
• Discovering Multi-Hardware Mobile Models via Architecture Search
• Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search
• Revision in Continuous Space: Unsupervised Text Style Transfer without Adversarial Learning
• LightSpeech: Lightweight and Fast Text to Speech with Neural Architecture Search
• Multi-Objective Neural Architecture Search Based on Diverse Structures and Adaptive Recommendation
• Disturbance-immune Weight Sharing for Neural Architecture Search
• Fisher Task Distance and Its Application in Neural Architecture Search
• Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
• On the Bounds of Function Approximations
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• Hyperparameter optimization with REINFORCE and Transformers
• Representation Sharing for Fast Object Detector Search and Beyond
• Interleaving Learning, with Application to Neural Architecture Search
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies
• G-DARTS-A: Groups of Channel Parallel Sampling with Attention
• MS-DARTS: Mean-Shift Based Differentiable Architecture Search
• Efficient Novelty-Driven Neural Architecture Search
• Towards Accurate and Compact Architectures via Neural Architecture Transformer
• Learning by Self-Explanation, with Application to Neural Architecture Search
• Zero-Cost Operation Scoring in Differentiable Architecture Search
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• EENA: Efficient Evolution of Neural Architecture
• Neural Architecture Search as Sparse Supernet
• Scaling Up Deep Neural Network Optimization for Edge Inference
• Disentangled Neural Architecture Search
• Searching for A Robust Neural Architecture in Four GPU Hours
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• Reconsidering CO2 emissions from Computer Vision
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• Learning from Mistakes based on Class Weighting with Application to Neural Architecture Search
• Evolving Neural Architecture Using One Shot Model
• Training BatchNorm Only in Neural Architecture Search and Beyond
• A Survey on Green Deep Learning
• Small-Group Learning, with Application to Neural Architecture Search
• Discretization-Aware Architecture Search
• Neural Architecture Search in Embedding Space
• AutoSNAP: Automatically Learning Neural Architectures for Instrument Pose Estimation