Learning Curves for Decision Making in Supervised Machine Learning: A Survey

Abstract

Learning curves are a concept from social sciences that has been adopted in the context of machine learning to assess the performance of a learning algorithm with respect to a certain resource, e.g., the number of training examples or the number of training iterations. Learning curves have important applications in several machine learning contexts, most notably in data acquisition, early stopping of model training, and model selection. For instance, learning curves can be used to model the performance of the combination of an algorithm and its hyperparameter configuration, providing insights into their potential suitability at an early stage and often expediting the algorithm selection process. Various learning curve models have been proposed to use learning curves for decision making. Some of these models answer the binary decision question of whether a given algorithm at a certain budget will outperform a certain reference performance, whereas more complex models predict the entire learning curve of an algorithm. We contribute a framework that categorises learning curve approaches using three criteria: the decision-making situation they address, the intrinsic learning curve question they answer and the type of resources they use. We survey papers from the literature and classify them into this framework.

Accepted in Machine Learning Journal

References in corpus (17)

• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
• Learning When Training Data are Costly: The Effect of Class Distribution on Tree Induction
• Sample Size Planning for Classification Models
• Non-stochastic Best Arm Identification and Hyperparameter Optimization
• Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets
• Multi-Objective Hyperparameter Optimization in Machine Learning -- An Overview
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
• Progressive Sampling-Based Bayesian Optimization for Efficient and Automatic Machine Learning Model Selection
• Robust Differentiable SVD
• Accelerating Neural Architecture Search using Performance Prediction
• Small Data, Big Decisions: Model Selection in the Small-Data Regime
• HPOBench: A Collection of Reproducible Multi-Fidelity Benchmark Problems for HPO
• Efficient Bayesian Learning Curve Extrapolation using Prior-Data Fitted Networks
• Recommending Training Set Sizes for Classification
• Inductive Transfer for Neural Architecture Optimization

Cited by in corpus (2)

• LHS in LHS: A new expansion strategy for Latin hypercube sampling in simulation design
• Beyond the Single-Best Model: Rashomon Partial Dependence Profile for Trustworthy Explanations in AutoML