A Probabilistic Method to Predict Classifier Accuracy on Larger Datasets given Small Pilot Data

Abstract

Practitioners building classifiers often start with a smaller pilot dataset and plan to grow to larger data in the near future. Such projects need a toolkit for extrapolating how much classifier accuracy may improve from a 2x, 10x, or 50x increase in data size. While existing work has focused on finding a single "best-fit" curve using various functional forms like power laws, we argue that modeling and assessing the uncertainty of predictions is critical yet has seen less attention. In this paper, we propose a Gaussian process model to obtain probabilistic extrapolations of accuracy or similar performance metrics as dataset size increases. We evaluate our approach in terms of error, likelihood, and coverage across six datasets. Though we focus on medical tasks and image modalities, our open source approach generalizes to any kind of classifier.

References in corpus (6)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Scaling Laws for Neural Language Models
• Deep Learning Scaling is Predictable, Empirically
• A New Semi-supervised Learning Benchmark for Classifying View and Diagnosing Aortic Stenosis from Echocardiograms
• Optimizing Data Collection for Machine Learning
• Learning Curves for Analysis of Deep Networks