On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice

Abstract

Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively.

Published in Neurocomputing (Elsevier's journal, Q1, IF: 5.779). Tutorial code has got 1000+ stars. Github link: https://github.com/LiYangHart/Hyperparameter-Optimization-of-Machine-Learning-Algorithms

References in corpus (17)

• Practical Bayesian Optimization of Machine Learning Algorithms
• A Tutorial on Principal Component Analysis
• MLlib: Machine Learning in Apache Spark
• Comparative Study of CNN and RNN for Natural Language Processing
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Hyperparameter Search in Machine Learning
• BayesOpt: A Bayesian Optimization Library for Nonlinear Optimization, Experimental Design and Bandits
• Systematic Ensemble Model Selection Approach for Educational Data Mining
• Automated Machine Learning: State-of-The-Art and Open Challenges
• Bayesian Optimization with Machine Learning Algorithms Towards Anomaly Detection
• Multi-split Optimized Bagging Ensemble Model Selection for Multi-class Educational Data Mining
• Efficient Hyperparameter Optimization of Deep Learning Algorithms Using Deterministic RBF Surrogates
• Combination of Hyperband and Bayesian Optimization for Hyperparameter Optimization in Deep Learning
• GPflowOpt: A Bayesian Optimization Library using TensorFlow
• Easy Hyperparameter Search Using Optunity
• DNS Typo-squatting Domain Detection: A Data Analytics & Machine Learning Based Approach
• Hyperparameter Optimization: A Spectral Approach

Cited by in corpus (63)

• Challenges in Deploying Machine Learning: a Survey of Case Studies
• MTH-IDS: A Multi-Tiered Hybrid Intrusion Detection System for Internet of Vehicles
• Multi-Stage Optimized Machine Learning Framework for Network Intrusion Detection
• A Lightweight Concept Drift Detection and Adaptation Framework for IoT Data Streams
• A Transfer Learning and Optimized CNN Based Intrusion Detection System for Internet of Vehicles
• Machine Learning Towards Intelligent Systems: Applications, Challenges, and Opportunities
• Review: Deep Learning in Electron Microscopy
• Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
• IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective
• On Leakage in Machine Learning Pipelines
• A Hybrid Method for Condition Monitoring and Fault Diagnosis of Rolling Bearings With Low System Delay
• Total Nitrogen Estimation in Agricultural Soils via Aerial Multispectral Imaging and LIBS
• An Optimal House Price Prediction Algorithm: XGBoost
• Multi-Perspective Content Delivery Networks Security Framework Using Optimized Unsupervised Anomaly Detection
• $\text{FastSVD-ML-ROM}$: A Reduced-Order Modeling Framework based on Machine Learning for Real-Time Applications
• Real-Time Adaptive Anomaly Detection in Industrial IoT Environments
• STC-IDS: Spatial-Temporal Correlation Feature Analyzing based Intrusion Detection System for Intelligent Connected Vehicles
• Enabling AutoML for Zero-Touch Network Security: Use-Case Driven Analysis
• Towards Zero Touch Networks: Cross-Layer Automated Security Solutions for 6G Wireless Networks
• Rolling the dice for better deep learning performance: A study of randomness techniques in deep neural networks
• Deep Learning Methods for Colloidal Silver Nanoparticle Concentration and Size Distribution Determination from UV-Vis Extinction Spectra
• Optimized Three Deep Learning Models Based-PSO Hyperparameters for Beijing PM2.5 Prediction
• Optimizing YOLOv7 for Semiconductor Defect Detection
• Automatic Assessment of Functional Movement Screening Exercises with Deep Learning Architectures
• Prediction of Permissioned Blockchain Performance for Resource Scaling Configurations
• Automating Outlier Detection via Meta-Learning
• Parallel Hyperparameter Optimization Of Spiking Neural Network
• System Architecture Optimization Strategies: Dealing with Expensive Hierarchical Problems
• Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble
• The Impact of COVID-19 Pandemic on Ridesourcing Services Differed Between Small Towns and Large Cities
• Finding Optimal Pathways in Chemical Reaction Networks Using Ising Machines
• Analyzing Geospatial and Socioeconomic Disparities in Breast Cancer Screening Among Populations in the United States: Machine Learning Approach
• Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters
• Improved companion mass limits for Sirius A with thermal infrared coronagraphy using a vector-apodizing phase plate and time-domain starlight-subtraction techniques
• Sentiment Analysis of Microblogging dataset on Coronavirus Pandemic
• Facilitating Database Tuning with Hyper-Parameter Optimization: A Comprehensive Experimental Evaluation
• REX: Revisiting Budgeted Training with an Improved Schedule
• Multi-Resolution Diffusion for Privacy-Sensitive Recommender Systems
• Distributed Nonconvex Optimization: Gradient-free Iterations and $\epsilon$-Globally Optimal Solution
• Evolutionary Augmentation Policy Optimization for Self-supervised Learning
• An Automated Data Engineering Pipeline for Anomaly Detection of IoT Sensor Data
• Iterative Feature Boosting for Explainable Speech Emotion Recognition
• Computational models of learning and synaptic plasticity
• Trend Extrapolation for Technology Forecasting: Leveraging LSTM Neural Networks for Trend Analysis of Space Exploration Vessels
• A distance for mixed-variable and hierarchical domains with meta variables
• A unified framework for detecting point and collective anomalies in operating system logs via collaborative transformers
• Scalable Acceleration for Classification-Based Derivative-Free Optimization
• An Unsupervised Deep Explainable AI Framework for Localization of Concurrent Replay Attacks in Nuclear Reactor Signals
• Detecting Network-based Internet Censorship via Latent Feature Representation Learning
• AI-driven Orchestration at Scale: Estimating Service Metrics on National-Wide Testbeds
• Practitioner Motives to Use Different Hyperparameter Optimization Methods
• Efficient safe learning for controller tuning with experimental validation
• Identification of Device Dependencies Using Link Prediction
• A Generative Framework for Low-Cost Result Validation of Machine Learning-as-a-Service Inference
• What augmentations are sensitive to hyper-parameters and why?
• Primitive Agentic First-Order Optimization
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Grammar-based evolutionary approach for automated workflow composition with domain-specific operators and ensemble diversity
• A general framework for adaptive nonparametric dimensionality reduction
• Social Media and Artificial Intelligence for Sustainable Cities and Societies: A Water Quality Analysis Use-case
• An Algorithm for Learning Smaller Representations of Models With Scarce Data
• AutoSurrogate: An LLM-Driven Multi-Agent Framework for Autonomous Construction of Deep Learning Surrogate Models in Subsurface Flow
• UniAutoML: A Human-Centered Framework for Unified Discriminative and Generative AutoML with Large Language Models