3 citations · 10 across the 13 of their papers we have counts for
13 papers
Open Problems in AI Risk Modeling: Insights from a Workshop on the Technical Foundations of AI Risk Modeling
Krystal Jackson, Deepika Raman, Jakub Kryś +8
We investigate the design of robust risk models to assess societal risks posed by advanced AI systems, an emerging area in AI governance. Many regulatory proposals increasingly req…
The 2026 Singapore Consensus on Global AI Safety Research Priorities
Stephen Casper, Oskar Galeev, Yoshua Bengio +117
Frontier AI capabilities and autonomy are advancing rapidly. A growing number of real-world incidents make a trusted AI ecosystem essential to embracing AI with confidence. The 202…
Open Problems in Frontier AI Risk Management
Marta Ziosi, Miro Plueckebaum, Stephen Casper +26
Frontier AI both amplifies existing risks and introduces qualitatively novel challenges. Not only is there a notable lack of stable scientific consensus resulting from the rapid pa…
Lessons from External Review of DeepMind's Scheming Inability Safety Case
Stephen Barrett, Francisco Javier Campos Zabala, Sean P. Fillingham +4
Safety cases for frontier AI systems should provide a convincing argument, supported by evidence, that the risk of harm is within an acceptable bound. When developers author their…
Frontier AI Auditing: Toward Rigorous Third-Party Assessment of Safety and Security Practices at Leading AI Companies
Miles Brundage, Noemi Dreksler, Aidan Homewood +45
We outline a vision for frontier AI auditing, which we define as rigorous third-party verification of frontier AI developers' safety and security claims, and evaluation of their sy…
Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse
Steve Barrett, Malcolm Murray, Otter Quarks +17
Advanced AI systems offer substantial benefits but also introduce risks. In 2025, AI-enabled cyber offense has emerged as a concrete example. This technical report applies a quanti…
A Methodology for Quantitative AI Risk Modeling
Malcolm Murray, Steve Barrett, Henry Papadatos +5
Although general-purpose AI systems offer transformational opportunities in science and industry, they simultaneously raise critical concerns about safety, misuse, and potential lo…
The Role of Risk Modeling in Advanced AI Risk Management
Chloé Touzet, Henry Papadatos, Malcolm Murray +6
Rapidly advancing artificial intelligence (AI) systems introduce novel, uncertain, and potentially catastrophic risks. Managing these risks requires a mature risk-management infras…
Evaluating AI Providers' Frontier Safety Frameworks
Lily Stelling, Malcolm Murray, Bruno Galizzi +3
Following the AI Seoul Summit in 2024, twelve AI companies published frontier AI safety frameworks (Frameworks) outlining their approaches to managing catastrophic risks from advan…
Evaluating the Goal-Directedness of Large Language Models
Tom Everitt, Cristina Garbacea, Alexis Bellot +4
To what extent do LLMs use their capabilities towards their given goal? We take this as a measure of their goal-directedness. We evaluate goal-directedness on tasks that require in…
Mapping AI Benchmark Data to Quantitative Risk Estimates Through Expert Elicitation
Malcolm Murray, Henry Papadatos, Otter Quarks +2
The literature and multiple experts point to many potential risks from large language models (LLMs), but there are still very few direct measurements of the actual harms posed. AI…
A Frontier AI Risk Management Framework: Bridging the Gap Between Current AI Practices and Established Risk Management
Simeon Campos, Henry Papadatos, Fabien Roger +3
The recent development of powerful AI systems has highlighted the need for robust risk management frameworks in the AI industry. Although companies have begun to implement safety f…
Linear Probe Penalties Reduce LLM Sycophancy
Henry Papadatos, Rachel Freedman
Large language models (LLMs) are often sycophantic, prioritizing agreement with their users over accurate or objective statements. This problematic behavior becomes more pronounced…