Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking
arXiv:2106.06052
Abstract
We introduce Dynaboard, an evaluation-as-a-service framework for hosting benchmarks and conducting holistic model comparison, integrated with the Dynabench platform. Our platform evaluates NLP models directly instead of relying on self-reported metrics or predictions on a single dataset. Under this paradigm, models are submitted to be evaluated in the cloud, circumventing the issues of reproducibility, accessibility, and backwards compatibility that often hinder benchmarking in NLP. This allows users to interact with uploaded models in real time to assess their quality, and permits the collection of additional metrics such as memory use, throughput, and robustness, which -- despite their importance to practitioners -- have traditionally been absent from leaderboards. On each task, models are ranked according to the Dynascore, a novel utility-based aggregation of these statistics, which users can customize to better reflect their preferences, placing more/less weight on a particular axis of evaluation or dataset. As state-of-the-art NLP models push the limits of traditional benchmarks, Dynaboard offers a standardized solution for a more diverse and comprehensive evaluation of model quality.
References in corpus (14)
- HuggingFace's Transformers: State-of-the-art Natural Language Processing
- Semantics derived automatically from language corpora contain human-like biases
- Bidirectional Attention Flow for Machine Comprehension
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
- Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
- NeurIPS 2020 EfficientQA Competition: Systems, Analyses and Lessons Learned
- The Ladder: A Reliable Leaderboard for Machine Learning Competitions
- The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
- Learning Gender-Neutral Word Embeddings
- HateCheck: Functional Tests for Hate Speech Detection Models
- ExplainaBoard: An Explainable Leaderboard for NLP
- Investigating Failures of Automatic Translation in the Case of Unambiguous Gender
- TextFlint: Unified Multilingual Robustness Evaluation Toolkit for Natural Language Processing