AutoML-Zero: Evolving Machine Learning Algorithms From Scratch

Abstract

Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.

Accepted for publication at the 37th International Conference on Machine Learning (ICML 2020). Near camera-ready version

References in corpus (24)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Practical Bayesian Optimization of Machine Learning Algorithms
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Neural Architecture Search with Reinforcement Learning
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• AutoAugment: Learning Augmentation Policies from Data
• Designing Neural Network Architectures using Reinforcement Learning
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Random Search and Reproducibility for Neural Architecture Search
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• DeepCoder: Learning to Write Programs
• Neural Optimizer Search with Reinforcement Learning
• Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Meta-Learning Update Rules for Unsupervised Representation Learning
• The Power of Normalization: Faster Evasion of Saddle Points
• ASAP: Architecture Search, Anneal and Prune
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Towards Synthesizing Complex Programs from Input-Output Examples
• Meta-learning curiosity algorithms
• Towards modular and programmable architecture search

Cited by in corpus (29)

• AutoML: A Survey of the State-of-the-Art
• Learning to Optimize: A Primer and A Benchmark
• Automated Evolutionary Approach for the Design of Composite Machine Learning Pipelines
• Evolving Normalization-Activation Layers
• Real-time 3D Nanoscale Coherent Imaging via Physics-aware Deep Learning
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• Handling Imbalanced Classification Problems With Support Vector Machines via Evolutionary Bilevel Optimization
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investment
• Evolving Reinforcement Learning Algorithms
• Qualities, challenges and future of genetic algorithms: a literature review
• Primer: Searching for Efficient Transformers for Language Modeling
• Meta Learning Backpropagation And Improving It
• Neural Architecture Search using Property Guided Synthesis
• Evolving to learn: discovering interpretable plasticity rules for spiking networks
• Symbolic Regression via Neural-Guided Genetic Programming Population Seeding
• Self-Organizing Intelligent Matter: A blueprint for an AI generating algorithm
• Towards Self-learning Edge Intelligence in 6G
• MPLP: Learning a Message Passing Learning Protocol
• Meta-Learning Bidirectional Update Rules
• Rethinking Neural Operations for Diverse Tasks
• Momentum Accelerates Evolutionary Dynamics
• There is No Such Thing as an "Index"! or: The next 500 Indexing Papers
• SIBILA: A novel interpretable ensemble of general-purpose machine learning models applied to medical contexts
• Formal Fields: A Framework to Automate Code Generation Across Domains
• Neural Architecture Search based on Cartesian Genetic Programming Coding Method
• Automated Decision-based Adversarial Attacks
• Bootstrapped Meta-Learning
• ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spaces