8 papers
CIRCLE: A Framework for Evaluating AI from a Real-World Lens
Reva Schwartz, Carina Westling, Morgan Briggs +12
This paper proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI's materialized outcomes in deployment.…
Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework
Cléa Chataigner, Rebecca Ma, Prakhar Ganesh +4
Large language models (LLMs) are highly sensitive to subtle changes in prompt phrasing, posing challenges for reliable auditing. Prior methods often apply unconstrained prompt para…
Systemizing Multiplicity: The Curious Case of Arbitrariness in Machine Learning
Prakhar Ganesh, Afaf Taik, Golnoosh Farnadi
Algorithmic modeling relies on limited information in data to extrapolate outcomes for unseen scenarios, often embedding an element of arbitrariness in its decisions. A perspective…
Balancing Profit and Fairness in Risk-Based Pricing Markets
Jesse Thibodeau, Hadi Nekoei, Afaf Taïk +2
Dynamic, risk-based pricing can systematically exclude vulnerable consumer groups from essential resources such as health insurance and consumer credit. We show that a regulator ca…
Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effects
Reva Schwartz, Rumman Chowdhury, Akash Kundu +17
Conventional AI evaluation approaches concentrated within the AI stack exhibit systemic limitations for exploring, navigating and resolving the human and societal factors that play…
Fairness in Federated Learning: Fairness for Whom?
Afaf Taik, Khaoula Chehbouni, Golnoosh Farnadi
Fairness in federated learning has emerged as a rapidly growing area of research, with numerous works proposing formal definitions and algorithmic interventions. Yet, despite this…