3 citations · 4 across the 5 of their papers we have counts for
6 papers
Transect: Retaining Observability for Long-Horizon LLM Agent Evaluations
Toby D. Pilditch, Konstantinos Voudouris, Alexandra Abbas +1
Frontier AI evaluations increasingly use open-ended, agentic, long-horizon tasks whose transcripts can span hundreds of pages of outputs and actions from complex multi-agent networ…
Knowing When to Stop: Bayesian Optimal Stopping for LLM Evaluations
Toby D. Pilditch
LLM evaluations often use fixed sampling budgets, testing every item the same number of times even after estimates are precise. We introduce optstop, a precision-based adaptive sto…
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations eithe…
Implications for Governance in Public Perceptions of Societal-scale AI Risks
Ross Gruetzemacher, Toby D. Pilditch, Huigang Liang +6
Amid growing concerns over AI's societal risks--ranging from civilizational collapse to misinformation and systemic bias--this study explores the perceptions of AI experts and the…
The Reasoning Under Uncertainty Trap: A Structural AI Risk
Toby D. Pilditch
This report examines a novel risk associated with current (and projected) AI tools. Making effective decisions about future actions requires us to reason under uncertainty (RUU), a…
An International Consortium for Evaluations of Societal-Scale Risks from Advanced AI
Ross Gruetzemacher, Alan Chan, Kevin Frazier +11
Given rapid progress toward advanced AI and risks from frontier AI systems (advanced AI systems pushing the boundaries of the AI capabilities frontier), the creation and implementa…