Hierarchical Representations for Efficient Architecture Search

Abstract

We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.

Accepted as a conference paper at ICLR 2018

References in corpus (3)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Architecture Search with Reinforcement Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks

Cited by in corpus (84)

• A Survey on Evolutionary Neural Architecture Search
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• SNAS: Stochastic Neural Architecture Search
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy
• D-VAE: A Variational Autoencoder for Directed Acyclic Graphs
• Auto-Keras: An Efficient Neural Architecture Search System
• Anomaly Detection for Water Treatment System based on Neural Network with Automatic Architecture Optimization
• MaskConnect: Connectivity Learning by Gradient Descent
• Language Models with Transformers
• Tuning Hyperparameters without Grad Students: Scalable and Robust Bayesian Optimisation with Dragonfly
• Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search
• Contrastive Self-supervised Neural Architecture Search
• Neural Graph Evolution: Towards Efficient Automatic Robot Design
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
• AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications
• NeurDB: An AI-powered Autonomous Data System
• PSViT: Better Vision Transformer via Token Pooling and Attention Sharing
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• BlockQNN: Efficient Block-wise Neural Network Architecture Generation
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• ReNAS:Relativistic Evaluation of Neural Architecture Search
• One-Shot Neural Architecture Search via Compressive Sensing
• Exploring Shared Structures and Hierarchies for Multiple NLP Tasks
• Hardware/Software Co-Exploration of Neural Architectures
• SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection
• Learnable Embedding Space for Efficient Neural Architecture Compression
• A Semi-Supervised Assessor of Neural Architectures
• Feature Partitioning for Efficient Multi-Task Architectures
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• Core-set Sampling for Efficient Neural Architecture Search
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• Neural Architecture Dilation for Adversarial Robustness
• Evolutionary Neural AutoML for Deep Learning
• DSRNA: Differentiable Search of Robust Neural Architectures
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• AutoML to Date and Beyond: Challenges and Opportunities
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Particle Swarm Optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks
• Trainless Model Performance Estimation for Neural Architecture Search
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• EDAS: Efficient and Differentiable Architecture Search
• Genetic Neural Architecture Search for automatic assessment of human sperm images
• Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?
• Neural Architecture Refinement: A Practical Way for Avoiding Overfitting in NAS
• Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks
• DNQ: Dynamic Network Quantization
• Effective, Efficient and Robust Neural Architecture Search
• Efficient Neural Network Compression
• Exploiting Operation Importance for Differentiable Neural Architecture Search
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• Device-Circuit-Architecture Co-Exploration for Computing-in-Memory Neural Accelerators
• G-DARTS-A: Groups of Channel Parallel Sampling with Attention
• Evolutionary Cell Aided Design for Neural Network Architectures
• ImmuNetNAS: An Immune-network approach for searching Convolutional Neural Network Architectures
• GEVO: GPU Code Optimization using Evolutionary Computation
• FP-NAS: Fast Probabilistic Neural Architecture Search
• Memory-Efficient Differentiable Transformer Architecture Search
• Genetic Network Architecture Search
• Hyperparameter optimization with REINFORCE and Transformers
• Real Image Super Resolution Via Heterogeneous Model Ensemble using GP-NAS
• FDNAS: Improving Data Privacy and Model Diversity in AutoML
• Efficient Sampling for Predictor-Based Neural Architecture Search
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• Hyperparameter Optimization in Neural Networks via Structured Sparse Recovery
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• Verisimilar Percept Sequences Tests for Autonomous Driving Intelligent Agent Assessment
• Neural Inheritance Relation Guided One-Shot Layer Assignment Search
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Network Architecture Search for Face Enhancement
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Mutually-aware Sub-Graphs Differentiable Architecture Search
• JUWELS Booster -- A Supercomputer for Large-Scale AI Research
• Edge-featured Graph Neural Architecture Search
• Max and Coincidence Neurons in Neural Networks
• VRoC: Variational Autoencoder-aided Multi-task Rumor Classifier Based on Text
• Accelerating Deep Learning Inference with Cross-Layer Data Reuse on GPUs