Showing 2024Show all
3 papers · 1 filter
cs.CL2024
Benchmarking LLMs' Judgments with No Gold Standard
Shengwei Xu, Yuxuan Lu, Grant Schoenebeck +1
We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by Large Language Models (LLMs), particularly in generati…
cs.CL2024
Eliciting Informative Text Evaluations with Large Language Models
Yuxuan Lu, Shengwei Xu, Yichi Zhang +2
Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar num…
cs.LG2024
Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms
Shengwei Xu, Yichi Zhang, Paul Resnick +1
Because high-quality data is like oxygen for AI systems, effectively eliciting information from crowdsourcing workers has become a first-order problem for developing high-performan…