35 citations · 94 across the 18 of their papers we have counts for
18 papers
The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits
Gemma Galdon Clavell, Pablo Accuosto, Usman Gohar
The rapid deployment of AI systems across high-stakes domains has created urgent demand for standardized evaluation, yet the field remains fragmented across competing risk taxonomi…
Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results
Jan Batzner, Sree Harsha Nelaturu, Damian Stachura +45
AI evaluations are widely used for testing and understanding progress. However, the diverse evaluators bring with them inconsistencies that challenge analysis and comparison. First…
Defeater Cards: Characterizing and Managing Safety Assurance Case Defeaters
Usman Gohar, Michael C. Hunter, Salil Purandare +3
Safety assurance cases provide structured justifications that safety-critical systems meet their safety requirements. Recently, the notion of defeaters has emerged as a rigorous me…
Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting
Avijit Ghosh, Anka Reuel, Jenny Chim +45
AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. The cost is interpretive: readers can…
International AI Safety Report 2026
Yoshua Bengio, Stephen Clare, Carina Prunkl +89
The International AI Safety Report 2026 synthesises the current scientific evidence on the capabilities, emerging risks, and safety of general-purpose AI systems. The report series…
When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
Mubashara Akhtar, Anka Reuel, Prajna Soni +34
Artificial intelligence benchmarks are an important mechanism to measure model progress and guide deployment decisions. However, benchmarks quickly "saturate", making it difficult…
Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations
Anka Reuel, Avijit Ghosh, Jenny Chim +32
Foundation models are increasingly central to high-stakes AI systems, and governance frameworks now depend on evaluations to assess their risks and capabilities. Although general c…
AILuminate: Introducing v1.0 of the AI Risk and Reliability Benchmark from MLCommons
Shaona Ghosh, Heather Frase, Adina Williams +99
The rapid advancement and deployment of AI systems have created an urgent need for standard safety-evaluation frameworks. This paper introduces AILuminate v1.0, the first comprehen…
A Family-Based Approach to Safety Cases for Controlled Airspaces in Small Uncrewed Aerial Systems
Michael C. Hunter, Usman Gohar, Myra B. Cohen +2
As small Uncrewed Aircraft Systems (sUAS) increasingly operate in the national airspace, safety concerns arise due to a corresponding rise in reported airspace violations and incid…
A Taxonomy of Real-World Defeaters in Safety Assurance Cases
Usman Gohar, Michael C. Hunter, Myra B. Cohen +1
The rise of cyber-physical systems in safety-critical domains calls for robust risk-evaluation frameworks. Assurance cases, often required by regulatory bodies, are a structured ap…
Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML
Prakhar Ganesh, Usman Gohar, Lu Cheng +1
With fairness concerns gaining significant attention in Machine Learning (ML), several bias mitigation techniques have been proposed, often compared against each other to find the…
CoDefeater: Using LLMs To Find Defeaters in Assurance Cases
Usman Gohar, Michael C. Hunter, Robyn R. Lutz +1
Constructing assurance cases is a widely used, and sometimes required, process toward demonstrating that safety-critical systems will operate safely in their planned environment. T…
Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges
Usman Gohar, Zeyu Tang, Jialu Wang +4
The widespread integration of Machine Learning systems in daily life, particularly in high-stakes domains, has raised concerns about the fairness implications. While prior works ha…
Introducing v0.5 of the AI Safety Benchmark from MLCommons
Bertie Vidgen, Adarsh Agrawal, Ahmed M. Ahmed +97
This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safe…
Towards Engineering Fair and Equitable Software Systems for Managing Low-Altitude Airspace Authorizations
Usman Gohar, Michael C. Hunter, Agnieszka Marczak-Czajka +3
Small Unmanned Aircraft Systems (sUAS) have gained widespread adoption across a diverse range of applications. This has introduced operational complexities within shared airspaces…
Evaluating the Social Impact of Generative AI Systems in Systems and Society
Irene Solaiman, Zeerak Talat, William Agnew +28
Generative AI systems across modalities, ranging from text (including code), image, audio, and video, have broad social impacts, but there is no official standard for means of eval…
A Survey on Intersectional Fairness in Machine Learning: Notions, Mitigation, and Challenges
Usman Gohar, Lu Cheng
The widespread adoption of Machine Learning systems, especially in more decision-critical applications such as criminal sentencing and bank loans, has led to increased concerns abo…
Towards Understanding Fairness and its Composition in Ensemble Machine Learning
Usman Gohar, Sumon Biswas, Hridesh Rajan
Machine Learning (ML) software has been widely adopted in modern society, with reported fairness implications for minority groups based on race, sex, age, etc. Many recent works ha…