4 papers
Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality
Xiaoyuan Zhu, Kimberly Le Truong, Riccardo Fogliato +8
As LLMs are deployed in high-stakes settings, users must judge the correctness of individual responses, often relying on model-generated justifications such as reasoning chains or…
Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas
Luke Guerdan, Justin Whitehouse, Kimberly Truong +2
As Generative AI (GenAI) systems see growing adoption, a key concern involves the external validity of evaluations, or the extent to which they generalize from lab-based to real-wo…
Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Styles
Kimberly Le Truong, Riccardo Fogliato, Hoda Heidari +1
Current benchmarks for evaluating Large Language Models (LLMs) often do not exhibit enough writing style diversity, with many adhering primarily to standardized conventions. Such b…
Toward Valid Measurement Of (Un)fairness For Generative AI: A Proposal For Systematization Through The Lens Of Fair Equality of Chances
Kimberly Le Truong, Annette Zimmermann, Hoda Heidari
Disparities in the societal harms and impacts of Generative AI (GenAI) systems highlight the critical need for effective unfairness measurement approaches. While numerous benchmark…