46 citations · 77 across the 10 of their papers we have counts for
10 papers
Consistency in Language Models: Current Landscape, Challenges, and Future Directions
Jekaterina Novikova, Carol Anderson, Borhane Blili-Hamelin +2
The hallmark of effective language use lies in consistency: expressing similar meanings in similar contexts and avoiding contradictions. While human communication naturally demonst…
In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI
Shayne Longpre, Kevin Klyman, Ruth E. Appel +31
The widespread deployment of general-purpose AI (GPAI) systems introduces significant new risks. Yet the infrastructure, practices, and norms for reporting flaws in GPAI systems re…
Stop treating `AGI' as the north-star goal of AI research
Borhane Blili-Hamelin, Christopher Graziul, Leif Hancox-Li +13
The AI research community plays a vital role in shaping the scientific, engineering, and societal goals of AI research. In this position paper, we argue that focusing on the highly…
Is ETHICS about ethics? Evaluating the ETHICS benchmark
Leif Hancox-Li, Borhane Blili-Hamelin
ETHICS is probably the most-cited dataset for testing the ethical capabilities of language models. Drawing on moral theory, psychology, and prompt evaluation, we interrogate the va…
Introducing v0.5 of the AI Safety Benchmark from MLCommons
Bertie Vidgen, Adarsh Agrawal, Ahmed M. Ahmed +97
This paper introduces v0.5 of the AI Safety Benchmark, which has been created by the MLCommons AI Safety Working Group. The AI Safety Benchmark has been designed to assess the safe…
A Safe Harbor for AI Evaluation and Red Teaming
Shayne Longpre, Sayash Kapoor, Kevin Klyman +20
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems. However, the terms of service and enforcement strategies used by promi…
Evolving AI Risk Management: A Maturity Model based on the NIST AI Risk Management Framework
Ravit Dotan, Borhane Blili-Hamelin, Ravi Madhavan +2
Researchers, government bodies, and organizations have been repeatedly calling for a shift in the responsible AI community from general principles to tangible and operationalizable…
A Framework for Assurance Audits of Algorithmic Systems
Khoa Lam, Benjamin Lange, Borhane Blili-Hamelin +3
An increasing number of regulations propose AI audits as a mechanism for achieving transparency and accountability for artificial intelligence (AI) systems. Despite some converging…
Unsocial Intelligence: an Investigation of the Assumptions of AGI Discourse
Borhane Blili-Hamelin, Leif Hancox-Li, Andrew Smart
Dreams of machines rivaling human intelligence have shaped the field of AI since its inception. Yet, the very meaning of human-level AI or artificial general intelligence (AGI) rem…
Making Intelligence: Ethical Values in IQ and ML Benchmarks
Borhane Blili-Hamelin, Leif Hancox-Li
In recent years, ML researchers have wrestled with defining and improving machine learning (ML) benchmarks and datasets. In parallel, some have trained a critical lens on the ethic…