Hyper-Parameter Optimization: A Review of Algorithms and Applications

Abstract

Since deep neural networks were developed, they have made huge contributions to everyday lives. Machine learning provides more rational advice than humans are capable of in almost every aspect of daily life. However, despite this achievement, the design and training of neural networks are still challenging and unpredictable procedures. To lower the technical thresholds for common users, automated hyper-parameter optimization (HPO) has become a popular topic in both academic and industrial areas. This paper provides a review of the most essential topics on HPO. The first section introduces the key hyper-parameters related to model training and structure, and discusses their importance and methods to define the value range. Then, the research focuses on major optimization algorithms and their applicability, covering their efficiency and accuracy especially for deep learning networks. This study next reviews major services and toolkits for HPO, comparing their support for state-of-the-art searching algorithms, feasibility with major deep learning frameworks, and extensibility for new modules designed by users. The paper concludes with problems that exist when HPO is applied to deep learning, a comparison between optimization algorithms, and prominent approaches for model evaluation with limited computational resources.

References in corpus (31)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Improving neural networks by preventing co-adaptation of feature detectors
• Practical Bayesian Optimization of Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• An overview of gradient descent optimization algorithms
• Neural Architecture Search with Reinforcement Learning
• How transferable are features in deep neural networks?
• Improved Regularization of Convolutional Neural Networks with Cutout
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Deep Learning using Rectified Linear Units (ReLU)
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Wide Residual Networks
• Searching for Activation Functions
• Tune: A Research Platform for Distributed Model Selection and Training
• Large Batch Training of Convolutional Networks
• Scalable Bayesian Optimization Using Deep Neural Networks
• Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
• A Tutorial on Bayesian Optimization
• Revisiting Small Batch Training for Deep Neural Networks
• Dealing with Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Dealing with Categorical and Integer-valued Variables in Bayesian Optimization with Gaussian Processes
• Neural Optimizer Search with Reinforcement Learning
• Meta-Learning: A Survey
• Freeze-Thaw Bayesian Optimization
• Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning
• An Entropy Search Portfolio for Bayesian Optimization
• TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries
• Oracle inequalities for computationally adaptive model selection

Cited by in corpus (36)

• AutoML: A Survey of the State-of-the-Art
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Toward Open-World Electroencephalogram Decoding Via Deep Learning: A Comprehensive Survey
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Optimized Three Deep Learning Models Based-PSO Hyperparameters for Beijing PM2.5 Prediction
• Batch Layer Normalization, A new normalization layer for CNNs and RNN
• Evolving Scientific Discovery by Unifying Data and Background Knowledge with AI Hilbert
• Game-Theoretic Model Predictive Control with Data-Driven Identification of Vehicle Model for Head-to-Head Autonomous Racing
• Revisiting Tensor Basis Neural Networks for Reynolds stress modeling: application to plane channel and square duct flows
• Lower Bounds and Accelerated Algorithms for Bilevel Optimization
• Validation Workflow for Machine Learning Interatomic Potentials for Complex Ceramics
• Automating Outlier Detection via Meta-Learning
• Machine learning-based classification for Single Photon Space Debris Light Curves
• Hyperparameter Optimization of Generative Adversarial Network Models for High-Energy Physics Simulations
• Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN
• Parallel Hyperparameter Optimization Of Spiking Neural Network
• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• Generative ensemble deep learning severe weather prediction from a deterministic convection-allowing model
• Intelligent Proactive Fault Tolerance at the Edge through Resource Usage Prediction
• Decoding Non-Linearity and Complexity: Deep Tabular Learning Approaches for Materials Science
• Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters
• ASFGNN: Automated Separated-Federated Graph Neural Network
• Predicting the Critical Number of Layers for Hierarchical Support Vector Regression
• An Overview and Solution for Democratizing AI Workflows at the Network Edge
• A Balanced Approach of Rapid Genetic Exploration and Surrogate Exploitation for Hyperparameter Optimization
• Label Propagation Techniques for Artifact Detection in Imbalanced Classes using Photoplethysmogram Signals
• Bilevel Optimization for Machine Learning: Algorithm Design and Convergence Analysis
• Hypernetwork-Based Augmentation
• Predicting Maximum Permitted Process Forces for Object Grasping and Manipulation Using a Deep Learning Regression Model
• On Compression Principle and Bayesian Optimization for Neural Networks
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Simple Hack for Transformers against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service
• MOFA: Modular Factorial Design for Hyperparameter Optimization
• A Survey on Green Deep Learning
• ChebLieNet: Invariant Spectral Graph NNs Turned Equivariant by Riemannian Geometry on Lie Groups
• AutoSmart: An Efficient and Automatic Machine Learning framework for Temporal Relational Data