51 citations · 78 across the 5 of their papers we have counts for
5 papers
DataPerf: Benchmarks for Data-Centric AI Development
Mark Mazumder, Colby Banbury, Xiaozhe Yao +42
Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithf…
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations
Ka Wong, Praveen Paritosh
Since the inception of crowdsourcing, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, ma…
Data Excellence for AI: Why Should You Care
Lora Aroyo, Matthew Lease, Praveen Paritosh +1
The efficacy of machine learning (ML) models depends on both algorithms and data. Training data defines what we want our models to learn, and testing data provides the means by whi…
Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability
Ka Wong, Praveen Paritosh, Lora Aroyo
We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a…
Metrology for AI: From Benchmarks to Instruments
Chris Welty, Praveen Paritosh, Lora Aroyo
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying…