54 citations · 181 across the 25 of their papers we have counts for
26 papers
AI Security Priorities: A Field-Wide Agenda
Gil Gekker, Rachel Steratore, Everett Smith +9
As AI systems are rapidly integrated into critical economic, governmental, and national security functions, the gap between AI adoption and AI security readiness continues to widen…
LinuxArena: A Control Setting for AI Agents in Live Production Software Environments
Tyler Tracy, Ram Potham, Nick Kuhn +31
We introduce LinuxArena, a control setting in which agents operate directly on live, multi-service production environments. LinuxArena contains 20 environments, 1,671 main tasks re…
Auditing Sabotage Bench: A Benchmark for Detecting and Fixing Research Sabotage in ML Codebases
Eric Gan, Aryan Bhatt, Buck Shlegeris +2
As AI systems are increasingly used to conduct research autonomously, misaligned systems could introduce subtle flaws that produce misleading results while evading detection. We in…
Evaluating Control Protocols for Untrusted AI Agents
Jon Kutasov, Chloe Loughridge, Yuqi Sun +4
As AI systems become more capable and widely deployed as agents, ensuring their safe operation becomes critical. AI control offers one approach to mitigating the risk from untruste…
SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents
Jonathan Kutasov, Yuqi Sun, Paul Colognese +9
As Large Language Models (LLMs) are increasingly deployed as autonomous agents in complex and long horizon settings, it is critical to evaluate their ability to sabotage users by p…
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
Tomek Korbak, Mikita Balesni, Elizabeth Barnes +38
AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known A…
The Singapore Consensus on Global AI Safety Research Priorities
Yoshua Bengio, Tegan Maharaj, Luke Ong +84
Rapidly improving AI capabilities and autonomy hold significant promise of transformation, but are also driving vigorous debate on how to ensure that AI is safe, i.e., trustworthy,…
Ctrl-Z: Controlling AI Agents via Resampling
Aryan Bhatt, Cody Rushing, Adam Kaufman +5
Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first contr…
How to evaluate control measures for LLM agents? A trajectory from today to superintelligence
Tomek Korbak, Mikita Balesni, Buck Shlegeris +1
As LLM agents grow more capable of causing harm autonomously, AI developers will rely on increasingly sophisticated control measures to prevent possibly misaligned agents from caus…
A sketch of an AI control safety case
Tomek Korbak, Joshua Clymer, Benjamin Hilton +2
As LLM agents gain a greater capacity to cause harm, AI developers might increasingly rely on control measures such as monitoring to justify that they are safe. We sketch how devel…
Alignment faking in large language models
Ryan Greenblatt, Carson Denison, Benjamin Wright +17
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its beha…
Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?
Alex Mallen, Charlie Griffin, Misha Wagner +2
An AI control protocol is a plan for usefully deploying AI systems that aims to prevent an AI from intentionally causing some unacceptable outcome. This paper investigates how well…
Adaptive Deployment of Untrusted LLMs Reduces Distributed Threats
Jiaxin Wen, Vivek Hebbar, Caleb Larson +9
As large language models (LLMs) become increasingly capable, it is prudent to assess whether safety measures remain effective even if LLMs intentionally try to bypass them. Previou…
Towards evaluations-based safety cases for AI scheming
Mikita Balesni, Marius Hobbhahn, David Lindner +13
We sketch how developers of frontier AI systems could construct a structured rationale -- a 'safety case' -- that an AI system is unlikely to cause catastrophic outcomes through sc…
Sabotage Evaluations for Frontier Models
Joe Benton, Misha Wagner, Eric Christiansen +13
Sufficiently capable models could subvert human oversight and decision-making in important contexts. For example, in the context of AI development, models could covertly sabotage e…
Games for AI Control: Models of Safety Evaluations of AI Deployment Protocols
Charlie Griffin, Louis Thomson, Buck Shlegeris +1
To evaluate the safety and usefulness of deployment protocols for untrusted AIs, AI Control uses a red-teaming exercise played between a protocol designer and an adversary. This pa…
Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models
Carson Denison, Monte MacDiarmid, Fazl Barez +11
In reinforcement learning, specification gaming occurs when AI systems learn undesired behaviors that are highly rewarded due to misspecified training goals. Specification gaming c…
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
Evan Hubinger, Carson Denison, Jesse Mu +36
Humans are capable of strategically deceptive behavior: behaving helpfully in most situations, but then behaving very differently in order to pursue alternative objectives when giv…
AI Control: Improving Safety Despite Intentional Subversion
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan +1
As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researcher…
Generalized Wick Decompositions
Chris MacLeod, Evgenia Nitishinskaya, Buck Shlegeris
We review the cumulant decomposition (a way of decomposing the expectation of a product of random variables (e.g. ) into a sum of terms corresponding to partitions…
Benchmarks for Detecting Measurement Tampering
Fabien Roger, Ryan Greenblatt, Max Nadeau +2
When training powerful AI systems to perform complex tasks, it may be challenging to provide training signals which are robust to optimization. One concern is \textit{measurement t…
Language models are better than humans at next-token prediction
Buck Shlegeris, Fabien Roger, Lawrence Chan +1
Current language models are considered to have sub-human capabilities at natural language tasks like question-answering or writing code. However, language models are not trained to…
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
Kevin Wang, Alexandre Variengien, Arthur Conmy +2
Research in mechanistic interpretability seeks to explain behaviors of machine learning models in terms of their internal components. However, most previous work either focuses on…
Polysemanticity and Capacity in Neural Networks
Adam Scherlis, Kshitij Sachan, Adam S. Jermyn +2
Individual neurons in neural networks often represent a mixture of unrelated features. This phenomenon, called polysemanticity, can make interpreting neural networks more difficult…