51 citations · 78 across the 5 of their papers we have counts for
Showing cs.AIShow all
2 papers · 1 filter
cs.AI2022
k-Rater Reliability: The Correct Unit of Reliability for Aggregated Human Annotations
Ka Wong, Praveen Paritosh
Since the inception of crowdsourcing, aggregation has been a common strategy for dealing with unreliable data. Aggregate ratings are more reliable than individual ones. However, ma…
cs.AI2019★ 20 cited
Metrology for AI: From Benchmarks to Instruments
Chris Welty, Praveen Paritosh, Lora Aroyo
In this paper we present the first steps towards hardening the science of measuring AI systems, by adopting metrology, the science of measurement and its application, and applying…