Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization

Abstract

Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

Changes: - Updated to JMLR version

Cited by in corpus (79)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• AutoML: A Survey of the State-of-the-Art
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Denoising IMU Gyroscopes with Deep Learning for Open-Loop Attitude Estimation
• Automated Machine Learning in Practice: State of the Art and Recent Results
• Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
• Stochastic analysis of heterogeneous porous material with modified neural architecture search (NAS) based physics-informed neural networks using transfer learning
• Autonomous synthesis of metastable materials
• IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective
• Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
• Multi-fidelity Bayesian Optimisation with Continuous Approximations
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• A machine learning approach for efficient uncertainty quantification using multiscale methods
• Escaping local minima with derivative-free methods: a numerical investigation
• Anisotropic 3D Multi-Stream CNN for Accurate Prostate Segmentation from Multi-Planar MRI
• Improvement of Performance in Freezing of Gait detection in Parkinsons Disease using Transformer networks and a single waist worn triaxial accelerometer
• Towards Green Automated Machine Learning: Status Quo and Future Directions
• Image-driven discriminative and generative machine learning algorithms for establishing microstructure-processing relationships
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• AutoPrognosis 2.0: Democratizing Diagnostic and Prognostic Modeling in Healthcare with Automated Machine Learning
• Learning the Effect of Registration Hyperparameters with HyperMorph
• Radial Basis Function Networks for Convolutional Neural Networks to Learn Similarity Distance Metric and Improve Interpretability
• Learning Curves for Decision Making in Supervised Machine Learning: A Survey
• HyperTendril: Visual Analytics for User-Driven Hyperparameter Optimization of Deep Neural Networks
• Machine Learning Emulation of Urban Land Surface Processes
• Similarity Learning based Few Shot Learning for ECG Time Series Classification
• VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
• 1D-CapsNet-LSTM: A Deep Learning-Based Model for Multi-Step Stock Index Forecasting
• Promoting Fairness through Hyperparameter Optimization
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models
• Otago Exercises Monitoring for Older Adults by a Single IMU and Hierarchical Machine Learning Models
• Automatic Meta-Path Discovery for Effective Graph-Based Recommendation
• Deep multi-task mining Calabi-Yau four-folds
• Hyper-Parameter Tuning for the (1+(λ,λ)) GA
• Transfer Learning for the Efficient Detection of COVID-19 from Smartphone Audio Data
• Deep Learning Development Environment in Virtual Reality
• U-Park: A User-Centric Smart Parking Recommendation System for Electric Shared Micromobility Services
• Parallel Architecture and Hyperparameter Search via Successive Halving and Classification
• Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning
• Impatient Bandits: Optimizing Recommendations for the Long-Term Without Delay
• Speedy Performance Estimation for Neural Architecture Search
• HyperSLICE: HyperBand optimized Spiral for Low-latency Interactive Cardiac Examination
• Model Parameter Identification via a Hyperparameter Optimization Scheme for Autonomous Racing Systems
• A Robust Learning Methodology for Uncertainty-aware Scientific Machine Learning models
• Artificial Neural Networks for Predicting Mechanical Properties of Crystalline Polyamide12 via Molecular Dynamics Simulations
• Sequential Gaussian Processes for Online Learning of Nonstationary Functions
• HYPPO: A Surrogate-Based Multi-Level Parallelism Tool for Hyperparameter Optimization
• Hyperparameter Optimization of Generative Adversarial Network Models for High-Energy Physics Simulations
• A hyperparameter-tuning approach to automated inverse planning
• Emulation Techniques for Scenario and Classical Control Design of Tokamak Plasmas
• Regularized boosting with an increasing coefficient magnitude stop criterion as meta-learner in hyperparameter optimization stacking ensemble
• Additive Tree-Structured Conditional Parameter Spaces in Bayesian Optimization: A Novel Covariance Function and a Fast Implementation
• Transfer Learning based Search Space Design for Hyperparameter Tuning
• SigOpt Mulch: An Intelligent System for AutoML of Gradient Boosted Trees
• On-device modeling of user's social context and familiar places from smartphone-embedded sensor data
• FReSCO: Flow Reconstruction and Segmentation for low latency Cardiac Output monitoring using deep artifact suppression and segmentation
• A Systematic Evaluation of Adversarial Attacks against Speech Emotion Recognition Models
• Discrete Simulation Optimization for Tuning Machine Learning Method Hyperparameters
• Deep Learning-Based Spatiotemporal Multi-Event Reconstruction for Delay Line Detectors
• Deep Semi-Supervised Learning for Time Series Classification
• What Makes a Top-Performing Precision Medicine Search Engine? Tracing Main System Features in a Systematic Way
• A Linear Programming Enhanced Genetic Algorithm for Hyperparameter Tuning in Machine Learning
• A Deep Neural Networks ensemble workflow from hyperparameter search to inference leveraging GPU clusters
• Two-step hyperparameter optimization method: Accelerating hyperparameter search by using a fraction of a training dataset
• CrossTrainer: Practical Domain Adaptation with Loss Reweighting
• The use of adversaries for optimal neural network training
• Generative Design of a Gas Turbine Combustor Using Invertible Neural Networks
• Automatic Machine Learning for Multi-Receiver CNN Technology Classifiers
• Early Diagnosis of Retinal Blood Vessel Damage via Deep Learning-Powered Collective Intelligence Models
• Additive Tree-Structured Covariance Function for Conditional Parameter Spaces in Bayesian Optimization
• Practitioner Motives to Use Different Hyperparameter Optimization Methods
• Boosting Resource-Constrained Federated Learning Systems with Guessed Updates
• Paramater Optimization for Manipulator Motion Planning using a Novel Benchmark Set
• Fast Unsupervised Deep Outlier Model Selection with Hypernetworks
• A Data-Driven Approach for Predicting Hydrodynamic Forces on Spherical Particles Using Volume Fraction Representations
• Neural Architecture Search for Sentence Classification with BERT
• AUTOKD: Automatic Knowledge Distillation Into A Student Architecture Family