Active Testing: Sample-Efficient Model Evaluation

Abstract

We introduce a new framework for sample-efficient model evaluation that we call active testing. While approaches like active learning reduce the number of labels needed for model training, existing literature largely ignores the cost of labeling test data, typically unrealistically assuming large test sets for model evaluation. This creates a disconnect to real applications, where test labels are important and just as expensive, e.g. for optimizing hyperparameters. Active testing addresses this by carefully selecting the test points to label, ensuring model evaluation is sample-efficient. To this end, we derive theoretically-grounded and intuitive acquisition strategies that are specifically tailored to the goals of active testing, noting these are distinct to those of active learning. As actively selecting labels introduces a bias; we further show how to remove this bias while reducing the variance of the estimator at the same time. Active testing is easy to implement and can be applied to any supervised machine learning method. We demonstrate its effectiveness on models including WideResNets and Gaussian processes on datasets including Fashion-MNIST and CIFAR-100.

arXiv commentPublished at the 38th International Conference on Machine Learning (ICML 2021)

References in corpus (15)

• Scikit-learn: Machine Learning in Python
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Improved Regularization of Convolutional Neural Networks with Cutout
• Weight Uncertainty in Neural Networks
• Bayesian Active Learning for Classification and Preference Learning
• BatchBALD: Efficient and Diverse Batch Acquisition for Deep Bayesian Active Learning
• Radial Bayesian Neural Networks: Beyond Discrete Support In Large-Scale Bayesian Deep Learning
• Super-Samples from Kernel Herding
• On Statistical Bias In Active Learning: How and When To Fix It
• UPAL: Unbiased Pool Based Active Learning
• Active Testing: An Efficient and Robust Framework for Estimating Accuracy
• Automated Model Selection with Bayesian Quadrature
• Active Bayesian Assessment for Black-Box Classifiers

Cited by in corpus (1)

• Sample Efficient Model Evaluation