19 citations · 69 across the 15 of their papers we have counts for
8 papers · 1 filter
The Limits of Inference Scaling Through Resampling
Benedikt Stroebl, Sayash Kapoor, Arvind Narayanan
Recent research has generated hope that inference scaling, such as resampling solutions until they pass verifiers like unit tests, could allow weaker models to match stronger ones.…
CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark
Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir +2
AI agents have the potential to aid users on a variety of consequential tasks, including conducting scientific research. To spur the development of useful agents, we need benchmark…
AI Agents That Matter
Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel +2
AI agents are an exciting new research direction, and agent development is driven by benchmarks. Our analysis of current agent benchmarks and evaluation practices reveals several s…
The Responsible Foundation Model Development Cheatsheet: A Review of Tools & Resources
Shayne Longpre, Stella Biderman, Alon Albalak +20
Foundation model development attracts a rapidly expanding body of contributors, scientists, and applications. To help shape responsible development practices, we introduce the Foun…
A Safe Harbor for AI Evaluation and Red Teaming
Shayne Longpre, Sayash Kapoor, Kevin Klyman +20
Independent evaluation and red teaming are critical for identifying the risks posed by generative AI systems. However, the terms of service and enforcement strategies used by promi…
On the Societal Impact of Open Foundation Models
Sayash Kapoor, Rishi Bommasani, Kevin Klyman +22
Foundation models are powerful technologies: how they are released publicly directly shapes their societal impact. In this position paper, we focus on open foundation models, defin…
Foundation Model Transparency Reports
Rishi Bommasani, Kevin Klyman, Shayne Longpre +5
Foundation models are critical digital technologies with sweeping societal impact that necessitates transparency. To codify how foundation model developers should provide transpare…
Promises and pitfalls of artificial intelligence for legal applications
Sayash Kapoor, Peter Henderson, Arvind Narayanan
Is AI set to redefine the legal profession? We argue that this claim is not supported by the current evidence. We dive into AI's increasingly prevalent roles in three types of lega…