10 citations · 15 across the 11 of their papers we have counts for
4 papers · 1 filter
Doubly-Robust LLM-as-a-Judge: Externally Valid Estimation with Imperfect Personas
Luke Guerdan, Justin Whitehouse, Kimberly Truong +2
As Generative AI (GenAI) systems see growing adoption, a key concern involves the external validity of evaluations, or the extent to which they generalize from lab-based to real-wo…
Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Tasks
Luke Guerdan, Devansh Saxena, Stevie Chancellor +2
Data scientists often formulate predictive modeling tasks involving fuzzy, hard-to-define concepts, such as the "authenticity" of student writing or the "healthcare need" of a pati…
Bridging Prediction and Intervention Problems in Social Systems
Lydia T. Liu, Inioluwa Deborah Raji, Angela Zhou +32
Many automated decision systems (ADS) are designed to solve prediction problems -- where the goal is to learn patterns from a sample of the population and apply them to individuals…
Validating LLM-as-a-Judge Systems under Rating Indeterminacy
Luke Guerdan, Solon Barocas, Kenneth Holstein +3
The LLM-as-a-judge paradigm, in which a judge LLM system replaces human raters in rating the outputs of other generative AI (GenAI) systems, plays a critical role in scaling and st…