HyperNOMAD: Hyperparameter optimization of deep neural networks using mesh adaptive direct search

Abstract

The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time consuming process that is often described as a "dark art". This explains the necessity of automating the calibration of these hyperparameters. Derivative-free optimization is a field that develops methods designed to optimize time consuming functions without relying on derivatives. This work introduces the HyperNOMAD package, an extension of the NOMAD software that applies the MADS algorithm [7] to simultaneously tune the hyperparameters responsible for both the architecture and the learning process of a deep neural network (DNN), and that allows for an important flexibility in the exploration of the search space by taking advantage of categorical variables. This new approach is tested on the MNIST and CIFAR-10 data sets and achieves results comparable to the current state of the art.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• A Survey on Deep Learning in Medical Image Analysis
• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search: A Survey
• An Introduction to Convolutional Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• CMA-ES for Hyperparameter Optimization of Deep Neural Networks
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Black-Box Optimization in Machine Learning with Trust Region Based Derivative Free Algorithm
• Learning to Learn Neural Networks

Cited by in corpus (4)

• Review: Deep Learning in Electron Microscopy
• Surrogate Optimization of Deep Neural Networks for Groundwater Predictions
• Scalable Derivative-Free Optimization for Nonlinear Least-Squares Problems
• Tuning a variational autoencoder for data accountability problem in the Mars Science Laboratory ground data system