1 citations · 1 across the 7 of their papers we have counts for
9 papers
Can This LLM Annotate My Data? Estimating Zero-Shot LLM Performance
Tong Lin, Grant Schoenebeck, Sabina Tomkins
Large language models (LLMs) are increasingly used in annotation tasks with varying success. While several studies have demonstrated the opportunities and shortcomings of LLMs for…
Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics
Shengwei Xu, Yuxuan Lu, Yifan Wu +2
Reference-based text evaluation metrics, which are widely used to assess natural language generation systems, score a candidate response by comparing it with a reference response.…
AnnotateThis: Analyzing a human-LLM system for annotating social media data with the concept of climate change mitigation pessimism
Zexuan Li, Derek Van Berkel, Ariel Hasell +3
Large language models (LLMs) are increasingly being integrated into research workflows. However, LLMs have been shown to struggle with difficult and nuanced concepts such as those…
Ad Insertion in LLM-Generated Responses
Shengwei Xu, Zhaohua Chen, Xiaotie Deng +2
Sustainable monetization of large language models (LLMs) remains a critical open challenge. Traditional search advertising, which relies on static keywords, fails to capture the fl…
Stochastically Dominant Peer Prediction
Yichi Zhang, Shengwei Xu, David Pennock +1
Eliciting reliable human feedback is essential for many machine learning tasks, such as learning from noisy labels and aligning AI systems with human preferences. Peer prediction m…
Benchmarking LLMs' Judgments with No Gold Standard
Shengwei Xu, Yuxuan Lu, Grant Schoenebeck +1
We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by Large Language Models (LLMs), particularly in generati…
Eliciting Informative Text Evaluations with Large Language Models
Yuxuan Lu, Shengwei Xu, Yichi Zhang +2
Peer prediction mechanisms motivate high-quality feedback with provable guarantees. However, current methods only apply to rather simple reports, like multiple-choice or scalar num…
Spot Check Equivalence: an Interpretable Metric for Information Elicitation Mechanisms
Shengwei Xu, Yichi Zhang, Paul Resnick +1
Because high-quality data is like oxygen for AI systems, effectively eliciting information from crowdsourcing workers has become a first-order problem for developing high-performan…
Eliciting Honest Information From Authors Using Sequential Review
Yichi Zhang, Grant Schoenebeck, Weijie Su
In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the i…