The Ladder: A Reliable Leaderboard for Machine Learning Competitions

Abstract

The organizer of a machine learning competition faces the problem of maintaining an accurate leaderboard that faithfully represents the quality of the best submission of each competing team. What makes this estimation problem particularly challenging is its sequential and adaptive nature. As participants are allowed to repeatedly evaluate their submissions on the leaderboard, they may begin to overfit to the holdout data that supports the leaderboard. Few theoretical results give actionable advice on how to design a reliable leaderboard. Existing approaches therefore often resort to poorly understood heuristics such as limiting the bit precision of answers and the rate of re-submission. In this work, we introduce a notion of "leaderboard accuracy" tailored to the format of a competition. We introduce a natural algorithm called "the Ladder" and demonstrate that it simultaneously supports strong theoretical guarantees in a fully adaptive model of estimation, withstands practical adversarial attacks, and achieves high utility on real submission files from an actual competition hosted by Kaggle. Notably, we are able to sidestep a powerful recent hardness result for adaptive risk estimation that rules out algorithms such as ours under a seemingly very similar notion of accuracy. On a practical note, we provide a completely parameter-free variant of our algorithm that can be deployed in a real competition with no tuning required whatsoever.

References in corpus (2)

• On the Generalization Properties of Differential Privacy
• More General Queries and Less Generalization Error in Adaptive Data Analysis

Cited by in corpus (31)

• Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
• Do ImageNet Classifiers Generalize to ImageNet?
• Neural Network Matrix Factorization
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Generalization in Adaptive Data Analysis and Holdout Reuse
• Learning with Differential Privacy: Stability, Learnability and the Sufficiency and Necessity of ERM Principle
• A Data Quality-Driven View of MLOps
• How much does your data exploration overfit? Controlling bias via information usage
• Privacy, ethics, and data access: A case study of the Fragile Families Challenge
• The Effect of Natural Distribution Shift on Question Answering Models
• Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
• Asynchronous Online Testing of Multiple Hypotheses
• Preserving Statistical Validity in Adaptive Data Analysis
• How to Host a Data Competition: Statistical Advice for Design and Analysis of a Data Competition
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• Continuous Integration of Machine Learning Models with ease.ml/ci: Towards a Rigorous Yet Practical Treatment
• The Benchmark Lottery
• PaloBoost: An Overfitting-robust TreeBoost with Out-of-Bag Sample Regularization Techniques
• Challenges in Bayesian Adaptive Data Analysis
• Quantitative Overfitting Management for Human-in-the-loop ML Application Development with ease.ml/meter
• Mitigating Bias in Adaptive Data Gathering via Differential Privacy
• Model Similarity Mitigates Test Set Overuse
• DeepNNK: Explaining deep models and their generalization using polytope interpolation
• Approval policies for modifications to Machine Learning-Based Software as a Medical Device: A study of bio-creep
• Natural Analysts in Adaptive Data Analysis
• Reducing overfitting in challenge-based competitions
• The Everlasting Database: Statistical Validity at a Fair Price
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• Optimal multiclass overfitting by sequence reconstruction from Hamming queries
• A Rademacher Complexity Based Method fo rControlling Power and Confidence Level in Adaptive Statistical Analysis
• Beyond the Leaderboard: Insight and Deployment Challenges to Address Research Problems