The Benchmark Lottery
arXiv:2107.07002
Abstract
The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of "a benchmark lottery" that describes the overall fragility of the ML benchmarking process. The benchmark lottery postulates that many factors, other than fundamental algorithmic superiority, may lead to a method being perceived as superior. On multiple benchmark setups that are prevalent in the ML community, we show that the relative performance of algorithms may be altered significantly simply by choosing different benchmark tasks, highlighting the fragility of the current paradigms and potential fallacious interpretation derived from benchmarking ML methods. Given that every benchmark makes a statement about what it perceives to be important, we argue that this might lead to biased progress in the community. We discuss the implications of the observed phenomena and provide recommendations on mitigating them using multiple machine learning domains and communities as use cases, including natural language processing, computer vision, information retrieval, recommender systems, and reinforcement learning.
References in corpus (42)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distributed Representations of Words and Phrases and their Compositionality
- Distilling the Knowledge in a Neural Network
- Playing Atari with Deep Reinforcement Learning
- Language Models are Few-Shot Learners
- Dueling Network Architectures for Deep Reinforcement Learning
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Emergence of Locomotion Behaviours in Rich Environments
- DeepMind Control Suite
- A large annotated corpus for learning natural language inference
- Massively Parallel Methods for Deep Reinforcement Learning
- Universal Transformers
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Deep Reinforcement Learning that Matters
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Challenges of Real-World Reinforcement Learning
- Revisiting ResNets: Improved Training and Scaling Strategies
- The Evolved Transformer
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Long Range Arena: A Benchmark for Efficient Transformers
- Wasserstein Auto-Encoders
- A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
- Do CIFAR-10 Classifiers Generalize to CIFAR-10?
- Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
- Neural Episodic Control
- Are we done with ImageNet?
- Cold Case: The Lost MNIST Digits
- The Ladder: A Reliable Leaderboard for Machine Learning Competitions
- How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
- Bringing the People Back In: Contesting Benchmark Machine Learning Datasets
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
- Re-evaluating Evaluation
- State of the Art Control of Atari Games Using Shallow Reinforcement Learning
- Are Pre-trained Convolutions Better than Pre-trained Transformers?
- Online and Offline Reinforcement Learning by Planning with a Learned Model
- Show Your Work: Improved Reporting of Experimental Results
- The advantages of multiple classes for reducing overfitting from test set reuse
- Utility is in the Eye of the User: A Critique of NLP Leaderboards
- Transferring Inductive Biases through Knowledge Distillation
- Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard
- How Robust are Model Rankings: A Leaderboard Customization Approach for Equitable Evaluation
- Rip van Winkle's Razor: A Simple Estimate of Overfit to Test Data