6 citations · 14 across the 11 of their papers we have counts for
15 papers
How to Have a Sensitive Debate: An Instance-Optimal Protocol for AI Debate
Jiawei Li, Zhiyang Xun, Lijie Chen +1
As powerful AI systems reach and sometimes surpass the abilities of human experts across a range of cognitively demanding tasks, the problem of accurate oversight and supervision o…
Debate Training Reduces Reward Hacking in RLAIF
Zachary Kenton, Lili Janzer, Rory Greig +8
We demonstrate that RL finetuning an LLM using debate, a two-player adversarial game between a generator and a critic adjudicated by a weaker LLM judge, reduces reward hacking comp…
Quantifying the Necessity of Chain of Thought through Opaque Serial Depth
Jonah Brown-Cohen, David Lindner, Rohin Shah
Large language models (LLMs) tend to externalize their reasoning in their chain of thought, making the chain of thought a good target for monitoring. This is partially an inherent…
Debate is efficient with your time
Jonah Brown-Cohen, Geoffrey Irving, Simon C. Marshall +3
AI safety via debate uses two competing models to help a human judge verify complex computational tasks. Previous work has established what problems debate can solve in principle,…
Avoiding Obfuscation with Prover-Estimator Debate
Jonah Brown-Cohen, Geoffrey Irving, Georgios Piliouras +3
Training powerful AI systems to exhibit desired behaviors hinges on the ability to provide accurate human supervision on increasingly complex tasks. A promising approach to this pr…
An Approach to Technical AGI Safety and Security
Rohin Shah, Alex Irpan, Alexander Matt Turner +27
Artificial General Intelligence (AGI) promises transformative benefits but also presents significant risks. We develop an approach to address the risk of harms consequential enough…
On scalable oversight with weak LLMs judging strong LLMs
Zachary Kenton, Noah Y. Siegel, János Kramár +8
Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a judge; consultancy, whe…
Scalable AI Safety via Doubly-Efficient Debate
Jonah Brown-Cohen, Geoffrey Irving, Georgios Piliouras
The emergence of pre-trained AI systems with powerful capabilities across a diverse and ever-increasing set of complex domains has raised a critical challenge for AI safety as task…
Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models
Dingli Yu, Simran Kaur, Arushi Gupta +3
With LLMs shifting their role from statistical modeling of language to serving as general-purpose AI agents, how should LLM evaluations change? Arguably, a key ability of an AI age…
Detecting Adversarial Directions in Deep Reinforcement Learning to Make Robust Decisions
Ezgi Korkmaz, Jonah Brown-Cohen
Learning in MDPs with highly complex state representations is currently possible due to multiple advancements in reinforcement learning algorithm design. However, this incline in c…
Faster Algorithms and Constant Lower Bounds for the Worst-Case Expected Error
Jonah Brown-Cohen
The study of statistical estimation without distributional assumptions on data values, but with knowledge of data collection methods was recently introduced by Chen, Valiant and Va…
Extended Formulation Lower Bounds for Refuting Random CSPs
Jonah Brown-Cohen, Prasad Raghavendra
Random constraint satisfaction problems (CSPs) such as random -SAT are conjectured to be computationally intractable. The average case hardness of random -SAT and other CSPs…
Formal Barriers to Longest-Chain Proof-of-Stake Protocols
Jonah Brown-Cohen, Arvind Narayanan, Christos-Alexandros Psomas +1
The security of most existing cryptocurrencies is based on a concept called Proof-of-Work, in which users must solve a computationally hard cryptopuzzle to authorize transactions (…
The matching problem has no small symmetric SDP
Gábor Braun, Jonah Brown-Cohen, Arefin Huq +5
Yannakakis showed that the matching problem does not have a small symmetric linear program. Rothvoß recently proved that any, not necessarily symmetric, linear program also has exp…
Combinatorial Optimization Algorithms via Polymorphisms
Jonah Brown-Cohen, Prasad Raghavendra
An elegant characterization of the complexity of constraint satisfaction problems has emerged in the form of the the algebraic dichotomy conjecture of [BKJ00]. Roughly speaking, th…