Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview

Abstract

Hyperparameter optimization constitutes a large part of typical modern machine learning workflows. This arises from the fact that machine learning methods and corresponding preprocessing steps often only yield optimal performance when hyperparameters are properly tuned. But in many applications, we are not only interested in optimizing ML pipelines solely for predictive accuracy; additional metrics or constraints must be considered when determining an optimal configuration, resulting in a multi-objective optimization problem. This is often neglected in practice, due to a lack of knowledge and readily available software implementations for multi-objective hyperparameter optimization. In this work, we introduce the reader to the basics of multi-objective hyperparameter optimization and motivate its usefulness in applied ML. Furthermore, we provide an extensive survey of existing optimization strategies, both from the domain of evolutionary algorithms and Bayesian optimization. We illustrate the utility of MOO in several specific ML applications, considering objectives such as operating conditions, prediction time, sparseness, fairness, interpretability and robustness.

Published at ACM TELO

References in corpus (10)

• On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice
• A Review of Evolutionary Multi-modal Multi-objective Optimization
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learning
• Evaluation Metrics for Unsupervised Learning Algorithms
• OpenBox: A Generalized Black-box Optimization Service
• Initial Design Strategies and their Effects on Sequential Model-Based Optimization
• Parallel Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints
• Neural Architecture Search as Multiobjective Optimization Benchmarks: Problem Formulation and Performance Assessment
• Multi-objective Hyper-parameter Optimization of Behavioral Song Embeddings

Cited by in corpus (4)

• Learning Curves for Decision Making in Supervised Machine Learning: A Survey
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• Generating Reliable Synthetic Clinical Trial Data: The Role of Hyperparameter Optimization and Domain Constraints
• Practitioner Motives to Use Different Hyperparameter Optimization Methods