4 papers
Justified or Just Convincing? Error Verifiability as a Dimension of LLM Quality
Xiaoyuan Zhu, Kimberly Le Truong, Riccardo Fogliato +8
As LLMs are deployed in high-stakes settings, users must judge the correctness of individual responses, often relying on model-generated justifications such as reasoning chains or…
Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas
Luke Guerdan, Justin Whitehouse, Kimberly Truong +2
As Generative AI (GenAI) systems see growing adoption, a key concern involves the external validity of evaluations, or the extent to which they generalize from lab-based to real-wo…
Toward Valid Measurement Of (Un)fairness For Generative AI: A Proposal For Systematization Through The Lens Of Fair Equality of Chances
Kimberly Le Truong, Annette Zimmermann, Hoda Heidari
Disparities in the societal harms and impacts of Generative AI (GenAI) systems highlight the critical need for effective unfairness measurement approaches. While numerous benchmark…
A structured regression approach for evaluating model performance across intersectional subgroups
Christine Herlihy, Kimberly Truong, Alexandra Chouldechova +1
Disaggregated evaluation is a central task in AI fairness assessment, where the goal is to measure an AI system's performance across different subgroups defined by combinations of…