Accounting for Variance in Machine Learning Benchmarks

Abstract

Strong empirical evidence that one machine-learning algorithm A outperforms another one B ideally calls for multiple trials optimizing the learning pipeline over sources of variation such as data sampling, data augmentation, parameter initialization, and hyperparameters choices. This is prohibitively expensive, and corners are cut to reach conclusions. We model the whole benchmarking process, revealing that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. We analyze the predominant comparison methods used today in the light of this variance. We show a counter-intuitive result that adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Building on these results, we study the error rate of detecting improvements, on five different deep-learning tasks/architectures. This study leads us to propose recommendations for performance comparisons.

Submitted to MLSys2021

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Auto-Encoding Variational Bayes
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Fully Convolutional Networks for Semantic Segmentation
• Deep Reinforcement Learning that Matters
• Self-training with Noisy Student improves ImageNet classification
• A Step Toward Quantifying Independently Reproducible Machine Learning Research

Cited by in corpus (7)

• How I failed machine learning in medical imaging -- shortcomings and recommendations
• Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoML
• A benchmark for computational analysis of animal behavior, using animal-borne tags
• A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels
• Estimating Motor Symptom Presence and Severity in Parkinson's Disease from Wrist Accelerometer Time Series using ROCKET and InceptionTime
• Mixture Manifold Networks: A Computationally Efficient Baseline for Inverse Modeling
• The Influence of Faulty Labels in Data Sets on Human Pose Estimation