CMA-ES for Hyperparameter Optimization of Deep Neural Networks

Abstract

Hyperparameters of deep neural networks are often optimized by grid search, random search or Bayesian optimization. As an alternative, we propose to use the Covariance Matrix Adaptation Evolution Strategy (CMA-ES), which is known for its state-of-the-art performance in derivative-free optimization. CMA-ES has some useful invariance properties and is friendly to parallel evaluations of solutions. We provide a toy example comparing CMA-ES and state-of-the-art Bayesian optimization algorithms for tuning the hyperparameters of a convolutional neural network for the MNIST dataset on 30 GPUs in parallel.

References in corpus (5)

• Practical Bayesian Optimization of Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• Scalable Bayesian Optimization Using Deep Neural Networks
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• Bayesian Optimization with Unknown Constraints

Cited by in corpus (42)

• A Survey on Evolutionary Neural Architecture Search
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks
• SMAC3: A Versatile Bayesian Optimization Package for Hyperparameter Optimization
• Weighted Random Search for CNN Hyperparameter Optimization
• A Genetic Programming Approach to Designing Convolutional Neural Network Architectures
• A Population-based Hybrid Approach to Hyperparameter Optimization for Neural Networks
• Learning Deep Morphological Networks with Neural Architecture Search
• CASI: A Convolutional Neural Network Approach for Shell Identification
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Generative Adversarial Networks for Financial Trading Strategies Fine-Tuning and Combination
• c-TPE: Tree-structured Parzen Estimator with Inequality Constraints for Expensive Hyperparameter Optimization
• How to pick the domain randomization parameters for sim-to-real transfer of reinforcement learning policies?
• Automated Self-Supervised Learning for Graphs
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• Multi-level CNN for lung nodule classification with Gaussian Process assisted hyperparameter optimization
• HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search
• Hyperboost: Hyperparameter Optimization by Gradient Boosting surrogate models
• Evolutionary Neural AutoML for Deep Learning
• Limited-Memory Matrix Adaptation for Large Scale Black-box Optimization
• Evolutionary Architecture Search For Deep Multitask Networks
• Automatic Configuration of Deep Neural Networks with EGO
• On Entropy Regularized Path Integral Control for Trajectory Optimization
• AIR5: Five Pillars of Artificial Intelligence Research
• Compiler-Level Matrix Multiplication Optimization for Deep Learning
• Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning
• Improving Evolutionary Strategies with Generative Neural Networks
• A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networks
• Evolutionary Variational Optimization of Generative Models
• Regularized Evolutionary Population-Based Training
• Selecting Data Adaptive Learner from Multiple Deep Learners using Bayesian Networks
• Information theoretic analysis of computational models as a tool to understand the neural basis of behaviors
• Self-building Neural Networks
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• RapidLayout: Fast Hard Block Placement of FPGA-optimized Systolic Arrays using Evolutionary Algorithms
• BGADAM: Boosting based Genetic-Evolutionary ADAM for Neural Network Optimization
• Efficient Automatic Meta Optimization Search for Few-Shot Learning
• Learning sparse transformations through backpropagation
• On tuning deep learning models: a data mining perspective
• Black-Box Optimization of Object Detector Scales
• Mirror Natural Evolution Strategies
• Genealogical Population-Based Training for Hyperparameter Optimization