Large-Scale Evolution of Image Classifiers

Abstract

Neural networks have proven effective at solving difficult problems but designing their architectures can be challenging, even for image classification problems alone. Our goal is to minimize human participation, so we employ evolutionary algorithms to discover such networks automatically. Despite significant computational requirements, we show that it is now possible to evolve models with accuracies within the range of those published in the last year. Specifically, we employ simple evolutionary techniques at unprecedented scales to discover models for the CIFAR-10 and CIFAR-100 datasets, starting from trivial initial conditions and reaching accuracies of 94.6% (95.6% for ensemble) and 77.0%, respectively. To do this, we use novel and intuitive mutation operators that navigate large search spaces; we stress that no human participation is required once evolution starts and that the output is a fully-trained model. Throughout this work, we place special emphasis on the repeatability of results, the variability in the outcomes and the computational requirements.

Accepted for publication at ICML 2017 (34th International Conference on Machine Learning)

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Practical Bayesian Optimization of Machine Learning Algorithms
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Architecture Search with Reinforcement Learning
• Striving for Simplicity: The All Convolutional Net
• Training Very Deep Networks
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• Learning both Weights and Connections for Efficient Neural Networks
• PlaNet - Photo Geolocation with Convolutional Neural Networks
• Deep Networks with Stochastic Depth
• Generative NeuroEvolution for Deep Learning
• Convolution by Evolution: Differentiable Pattern Producing Networks

Cited by in corpus (223)

• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• AutoML: A Survey of the State-of-the-Art
• Automatically designing CNN architectures using genetic algorithm for image classification
• A Survey on Evolutionary Neural Architecture Search
• NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing
• Towards a Human-like Open-Domain Chatbot
• Understanding and Robustifying Differentiable Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• A Survey on Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Neural Architecture Transfer
• Differentiable Quantum Architecture Search
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
• Graph HyperNetworks for Neural Architecture Search
• NAS evaluation is frustratingly hard
• Automated Machine Learning in Practice: State of the Art and Recent Results
• Coin.AI: A Proof-of-Useful-Work Scheme for Blockchain-based Distributed Deep Learning
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Neural Predictor based Quantum Architecture Search
• Auto-Keras: An Efficient Neural Architecture Search System
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Automated Machine Learning on Graphs: A Survey
• EEEA-Net: An Early Exit Evolutionary Neural Architecture Search
• ASAP: Architecture Search, Anneal and Prune
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• Efficient Neural Architecture Transformation Searchin Channel-Level for Object Detection
• Learning Generalisable Omni-Scale Representations for Person Re-Identification
• Stabilizing DARTS with Amended Gradient Estimation on Architectural Parameters
• Neural Architecture Search For Keyword Spotting
• Searching Central Difference Convolutional Networks for Face Anti-Spoofing
• ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding
• Deeper Insights into Weight Sharing in Neural Architecture Search
• sharpDARTS: Faster and More Accurate Differentiable Architecture Search
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Reinforced Evolutionary Neural Architecture Search
• Finding Competitive Network Architectures Within a Day Using UCT
• Rethinking Architecture Selection in Differentiable NAS
• Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization
• CryptoNAS: Private Inference on a ReLU Budget
• GOLD-NAS: Gradual, One-Level, Differentiable
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Triple-level Model Inferred Collaborative Network Architecture for Video Deraining
• EvoPose2D: Pushing the Boundaries of 2D Human Pose Estimation using Accelerated Neuroevolution with Weight Transfer
• Constrained Design of Deep Iris Networks
• Latency-Aware Differentiable Neural Architecture Search
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Evolving Rewards to Automate Reinforcement Learning
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining
• CurveLane-NAS: Unifying Lane-Sensitive Architecture Search and Adaptive Point Blending
• AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations
• DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation
• Model-based Asynchronous Hyperparameter and Neural Architecture Search
• Differentiable Neural Input Search for Recommender Systems
• Evolving Reinforcement Learning Algorithms
• Neural Predictor for Neural Architecture Search
• Evolving Robust Neural Architectures to Defend from Adversarial Attacks
• MANAS: Multi-Agent Neural Architecture Search
• Automated Concatenation of Embeddings for Structured Prediction
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Neural Architecture Generator Optimization
• Learned Low Precision Graph Neural Networks
• Multi-Objective Evolutionary Design of Deep Convolutional Neural Networks for Image Classification
• Real-time Federated Evolutionary Neural Architecture Search
• Bag of Baselines for Multi-objective Joint Neural Architecture Search and Hyperparameter Optimization
• Convolution Neural Network Architecture Learning for Remote Sensing Scene Classification
• SS-Auto: A Single-Shot, Automatic Structured Weight Pruning Framework of DNNs with Ultra-High Efficiency
• SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection
• Trilevel Neural Architecture Search for Efficient Single Image Super-Resolution
• Speedy Performance Estimation for Neural Architecture Search
• FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
• ClassyTune: A Performance Auto-Tuner for Systems in the Cloud
• Searching for Efficient Multi-Stage Vision Transformers
• Analyzing and Mitigating Interference in Neural Architecture Search
• Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• A Semi-Supervised Assessor of Neural Architectures
• Core-set Sampling for Efficient Neural Architecture Search
• GeneCAI: Genetic Evolution for Acquiring Compact AI
• Self-supervised Neural Architecture Search
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• Sampled Training and Node Inheritance for Fast Evolutionary Neural Architecture Search
• A Variational-Sequential Graph Autoencoder for Neural Architecture Performance Prediction
• Fast Task-Aware Architecture Inference
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• AutoCaption: Image Captioning with Neural Architecture Search
• Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search
• Improving One-shot NAS by Suppressing the Posterior Fading
• Standing on the Shoulders of Giants: Hardware and Neural Architecture Co-Search with Hot Start
• Automatic Model Selection for Neural Networks
• Neural Architecture Optimization with Graph VAE
• SwiftNet: Using Graph Propagation as Meta-knowledge to Search Highly Representative Neural Architectures
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• Flexible Multi-task Networks by Learning Parameter Allocation
• A Novel Framework for Neural Architecture Search in the Hill Climbing Domain
• Neural Architecture Search for Deep Image Prior
• MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation
• Circumventing Outliers of AutoAugment with Knowledge Distillation
• flexgrid2vec: Learning Efficient Visual Representations Vectors
• Neuroevolution in Deep Learning: The Role of Neutrality
• Full-Cycle Energy Consumption Benchmark for Low-Carbon Computer Vision
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Rethinking Graph Neural Architecture Search from Message-passing
• A Survey On Anti-Spoofing Methods For Face Recognition with RGB Cameras of Generic Consumer Devices
• Powering One-shot Topological NAS with Stabilized Share-parameter Proxy
• AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks
• ImmuNeCS: Neural Committee Search by an Artificial Immune System
• Self-Adaptive 2D-3D Ensemble of Fully Convolutional Networks for Medical Image Segmentation
• PV-NAS: Practical Neural Architecture Search for Video Recognition
• Learning Strict Identity Mappings in Deep Residual Networks
• Evolving Search Space for Neural Architecture Search
• Accuracy Prediction with Non-neural Model for Neural Architecture Search
• Neural Architecture Search by Estimation of Network Structure Distributions
• Darts-Conformer: Towards Efficient Gradient-Based Neural Architecture Search For End-to-End ASR
• InstaNAS: Instance-aware Neural Architecture Search
• Inductive Transfer for Neural Architecture Optimization
• Efficient Decoupled Neural Architecture Search by Structure and Operation Sampling
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• Are Labels Necessary for Neural Architecture Search?
• Integrating Large Circular Kernels into CNNs through Neural Architecture Search
• Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation
• A Generalized Framework for Population Based Training
• Particle Swarm Optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks
• Controlling Model Complexity in Probabilistic Model-Based Dynamic Optimization of Neural Network Structures
• Exploiting Channel Similarity for Accelerating Deep Convolutional Neural Networks
• How can AI Automate End-to-End Data Science?
• NAS-Count: Counting-by-Density with Neural Architecture Search
• A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• Epigenetic evolution of deep convolutional models
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
• BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search
• PK-GCN: Prior Knowledge Assisted Image Classification using Graph Convolution Networks
• Learning Navigation Behaviors End-to-End with AutoRL
• Evolutionary Construction of Convolutional Neural Networks
• Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
• Generic Neural Architecture Search via Regression
• Training convolutional neural networks with cheap convolutions and online distillation
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• NAS-HPO-Bench-II: A Benchmark Dataset on Joint Optimization of Convolutional Neural Network Architecture and Training Hyperparameters
• Fast Hardware-Aware Neural Architecture Search
• RAPDARTS: Resource-Aware Progressive Differentiable Architecture Search
• Intelligence, physics and information -- the tradeoff between accuracy and simplicity in machine learning
• On the Bounds of Function Approximations
• Differentiable Learning-to-Group Channels via Groupable Convolutional Neural Networks
• Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
• Evolutionary Variational Optimization of Generative Models
• Tiny Adversarial Mulit-Objective Oneshot Neural Architecture Search
• Adaptive Bayesian Linear Regression for Automated Machine Learning
• EPNAS: Efficient Progressive Neural Architecture Search
• Heuristic Rank Selection with Progressively Searching Tensor Ring Network
• ImmuNetNAS: An Immune-network approach for searching Convolutional Neural Network Architectures
• Differentiable Neural Architecture Learning for Efficient Neural Network Design
• NASIB: Neural Architecture Search withIn Budget
• Accelerating Reinforcement Learning with a Directional-Gaussian-Smoothing Evolution Strategy
• NAS-TC: Neural Architecture Search on Temporal Convolutions for Complex Action Recognition
• Single-level Optimization For Differential Architecture Search
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Automatic Inference of Cross-modal Connection Topologies for X-CNNs
• GEVO: GPU Code Optimization using Evolutionary Computation
• Hyperparameter optimization with REINFORCE and Transformers
• Evolutionary Cell Aided Design for Neural Network Architectures
• Representation Sharing for Fast Object Detector Search and Beyond
• DarwinML: A Graph-based Evolutionary Algorithm for Automated Machine Learning
• Evolving neural networks to follow trajectories of arbitrary complexity
• EIGEN: Ecologically-Inspired GENetic Approach for Neural Network Structure Searching from Scratch
• iDARTS: Improving DARTS by Node Normalization and Decorrelation Discretization
• Accelerating Entrepreneurial Decision-Making Through Hybrid Intelligence
• Rapid Model Architecture Adaption for Meta-Learning
• Searching for A Robust Neural Architecture in Four GPU Hours
• BiX-NAS: Searching Efficient Bi-directional Architecture for Medical Image Segmentation
• ASCAI: Adaptive Sampling for acquiring Compact AI
• Ranking architectures using meta-learning
• GCCN: Global Context Convolutional Network
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning
• Two Novel Performance Improvements for Evolving CNN Topologies
• Direct Federated Neural Architecture Search
• Signature-Graph Networks
• EENA: Efficient Evolution of Neural Architecture
• ADWPNAS: Architecture-Driven Weight Prediction for Neural Architecture Search
• Rapid Elastic Architecture Search under Specialized Classes and Resource Constraints
• Adjoined Networks: A Training Paradigm with Applications to Network Compression
• Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
• FLASH: Fast Neural Architecture Search with Hardware Optimization
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• Differentiable Feature Aggregation Search for Knowledge Distillation
• AdvantageNAS: Efficient Neural Architecture Search with Credit Assignment
• Unchain the Search Space with Hierarchical Differentiable Architecture Search
• Edge-featured Graph Neural Architecture Search
• An Analysis of Super-Net Heuristics in Weight-Sharing NAS
• Layout Design for Intelligent Warehouse by Evolution with Fitness Approximation
• PEng4NN: An Accurate Performance Estimation Engine for Efficient Automated Neural Network Architecture Search
• Hardware-aware Real-time Myocardial Segmentation Quality Control in Contrast Echocardiography
• Probabilistic solution of chaotic dynamical system inverse problems using Bayesian Artificial Neural Networks
• AmoebaContact and GDFold: a new pipeline for rapid prediction of protein structures
• Blending Pruning Criteria for Convolutional Neural Networks
• AutoBSS: An Efficient Algorithm for Block Stacking Style Search
• Surrogate-assisted Particle Swarm Optimisation for Evolving Variable-length Transferable Blocks for Image Classification
• ConAM: Confidence Attention Module for Convolutional Neural Networks
• Single-DARTS: Towards Stable Architecture Search
• Evolving Neural Architecture Using One Shot Model
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family
• Revisiting Neural Architecture Search
• Architecture Agnostic Neural Networks
• Universality of Gradient Descent Neural Network Training
• Guided Evolution for Neural Architecture Search
• Deep Learning and Open Set Malware Classification: A Survey
• Nucleus Neural Network: A Data-driven Self-organized Architecture
• Efficient Transfer Learning via Joint Adaptation of Network Architecture and Weight