Simple And Efficient Architecture Search for Convolutional Neural Networks

Abstract

Neural networks have recently had a lot of success for many tasks. However, neural network architectures that perform well are still typically designed manually by experts in a cumbersome trial-and-error process. We propose a new method to automatically search for well-performing CNN architectures based on a simple hill climbing procedure whose operators apply network morphisms, followed by short optimization runs by cosine annealing. Surprisingly, this simple method yields competitive results, despite only requiring resources in the same order of magnitude as training a single network. E.g., on CIFAR-10, our method designs and trains networks with an error rate below 6% in only 12 hours on a single GPU; training for one day reduces this error further, to almost 5%.

Under review as a conference paper at ICLR 2018

References in corpus (5)

• Neural Architecture Search with Reinforcement Learning
• Densely Connected Convolutional Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Shake-Shake regularization
• Snapshot Ensembles: Train 1, get M for free

Cited by in corpus (46)

• AutoAugment: Learning Augmentation Policies from Data
• A Survey on Evolutionary Neural Architecture Search
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Neuroevolution in Deep Neural Networks: Current Trends and Future Challenges
• Automated Machine Learning in Practice: State of the Art and Recent Results
• D-VAE: A Variational Autoencoder for Directed Acyclic Graphs
• Auto-Keras: An Efficient Neural Architecture Search System
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Language Models with Transformers
• Contrastive Self-supervised Neural Architecture Search
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Manas: Mining Software Repositories to Assist AutoML
• Evolutionary-Neural Hybrid Agents for Architecture Search
• Multi-level CNN for lung nodule classification with Gaussian Process assisted hyperparameter optimization
• Designing deep neural networks for driver intention recognition
• Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
• An Introduction to Neural Architecture Search for Convolutional Networks
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning
• A Novel Framework for Neural Architecture Search in the Hill Climbing Domain
• Prune and Replace NAS
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Fast Neural Architecture Construction using EnvelopeNets
• Neural Architecture Search by Estimation of Network Structure Distributions
• ESAI: Efficient Split Artificial Intelligence via Early Exiting Using Neural Architecture Search
• VINNAS: Variational Inference-based Neural Network Architecture Search
• Deep Neural Architecture Search with Deep Graph Bayesian Optimization
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Particle Swarm Optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks
• Intra-Ensemble in Neural Networks
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• DeepSwarm: Optimising Convolutional Neural Networks using Swarm Intelligence
• Exploring the Loss Landscape in Neural Architecture Search
• On the Exploitation of Neuroevolutionary Information: Analyzing the Past for a More Efficient Future
• The Elastic Lottery Ticket Hypothesis
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• AIPerf: Automated machine learning as an AI-HPC benchmark
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Evolutionary Algorithm Enhanced Neural Architecture Search for Text-Independent Speaker Verification
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• Making Differentiable Architecture Search less local
• "You might also like this model": Data Driven Approach for Recommending Deep Learning Models for Unknown Image Datasets
• Multi-headed Neural Ensemble Search