17 citations · 27 across the 6 of their papers we have counts for
7 papers
A Framework for Generating Valid Context-Specific Benchmarks through Expert Guidance
Kimberly Le Truong, Nari Johnson, Anna Kawakami +1
This paper presents an end-to-end approach for generating context-specific large language model (LLM) benchmark datasets by combining expert input with synthetic data generation. E…
Evaluating AI-Generated Images of Cultural Artifacts with Community-Informed Rubrics
Nari Johnson, Deepthi Sudharsan, Hamna +7
Measurement is essential to improving AI performance and mitigating harms for marginalized groups. As generative AI systems are rapidly deployed across geographies and contexts, AI…
Disclosure or Marketing? Analyzing the Efficacy of Vendor Self-reports for Vetting Public-sector AI
Blaine Kuehnert, Nari Johnson, Ravit Dotan +1
Documentation-based disclosure has become a central governance strategy for responsible AI, particularly in public-sector procurement. Tools such as model cards, datasheets, and AI…
Provocation: Who benefits from "inclusion" in Generative AI?
Samantha Dalal, Siobhan Mackenzie Hall, Nari Johnson
The demands for accurate and representative generative AI systems means there is an increased demand on participatory evaluation structures. While these participatory structures ar…
Legacy Procurement Practices Shape How U.S. Cities Govern AI: Understanding Government Employees' Practices, Challenges, and Needs
Nari Johnson, Elise Silva, Harrison Leon +4
Most AI tools adopted by governments are not developed internally, but instead are acquired from third-party vendors in a process called public procurement. In this paper, we condu…
The Fall of an Algorithm: Characterizing the Dynamics Toward Abandonment
Nari Johnson, Sanika Moharana, Christina N. Harrington +3
As more algorithmic systems have come under scrutiny for their potential to inflict societal harms, an increasing number of organizations that hold power over harmful algorithms ha…
Assessing AI Impact Assessments: A Classroom Study
Nari Johnson, Hoda Heidari
Artificial Intelligence Impact Assessments ("AIIAs"), a family of tools that provide structured processes to imagine the possible impacts of a proposed AI system, have become an in…