24 citations · 37 across the 15 of their papers we have counts for
16 papers
Red-Teaming Auto Mode: Improving Blocking Classifiers Against Malign Coding Agents
Alex Remedios, Simon Storf, Fabien Roger +1
To keep coding agents from going off the rails, production systems now review each proposed action with a blocking monitor that can reject it before it runs (Auto Mode in Claude Co…
Overthinking: Amplifying Reasoning Weights to Extract Learned Secrets
Jack Hopkins, Dipika Khullar, Fabien Roger
Black box auditing of language models is an essential pre-deployment tool, but it may miss subtle forms of misalignment and hidden information. To better elicit hidden information…
Narrow Secret Loyalty Dodges Black-Box Audits
Alfie Lamerton, Fabien Roger
Recent work identifies secret loyalties as a distinct threat from standard backdoors. A secret loyalty causes a model to covertly advance the interests of a specific principal whil…
SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
Elle Najt, Colin Toft, Tyler Tracy +2
Since autonomous coding agents generate complex behaviors at high-volume, we may want to use other LLMs to monitor actions to reduce the risk from dangerous misaligned behavior. To…
Classifier Context Rot: Monitor Performance Degrades with Context Length
Sam Martin, Fabien Roger
Monitoring coding agents for dangerous behavior using language models requires classifying transcripts that often exceed 500K tokens, but prior agent monitoring benchmarks rarely c…
How Useful Is Cross-Domain Generalization for Training LLM Monitors?
Sam Martin, Fabien Roger
Using prompted language models as classifiers enables classification in domains with limited training data, but misses some of the robustness and performance benefits that fine-tun…
Self-Attribution Bias: When AI Monitors Go Easy on Themselves
Dipika Khullar, Jack Hopkins, Rowan Wang +1
Agentic systems increasingly rely on language models to monitor their own behavior. For example, coding agents may self critique generated code for pull request approval or assess…
Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
Tomek Korbak, Mikita Balesni, Elizabeth Barnes +38
AI systems that "think" in human language offer a unique opportunity for AI safety: we can monitor their chains of thought (CoT) for the intent to misbehave. Like all other known A…
Why Do Some Language Models Fake Alignment While Others Don't?
Abhay Sheshadri, John Hughes, Julian Michael +4
Alignment faking in large language models presented a demonstration of Claude 3 Opus and Claude 3.5 Sonnet selectively complying with a helpful-only training objective to prevent m…
Alignment faking in large language models
Ryan Greenblatt, Carson Denison, Benjamin Wright +17
We present a demonstration of a large language model engaging in alignment faking: selectively complying with its training objective in training to prevent modification of its beha…
Stress-Testing Capability Elicitation With Password-Locked Models
Ryan Greenblatt, Fabien Roger, Dmitrii Krasheninnikov +1
To determine the safety of large language models (LLMs), AI developers must be able to assess their dangerous capabilities. But simple prompting strategies often fail to elicit an…
AI Control: Improving Safety Despite Intentional Subversion
Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan +1
As large language models (LLMs) become more powerful and are deployed more autonomously, it will be increasingly important to prevent them from causing harmful outcomes. Researcher…
Preventing Language Models From Hiding Their Reasoning
Fabien Roger, Ryan Greenblatt
Large language models (LLMs) often benefit from intermediate steps of reasoning to generate answers to complex problems. When these intermediate steps of reasoning are used to moni…
Benchmarks for Detecting Measurement Tampering
Fabien Roger, Ryan Greenblatt, Max Nadeau +2
When training powerful AI systems to perform complex tasks, it may be challenging to provide training signals which are robust to optimization. One concern is \textit{measurement t…
Large Language Models Sometimes Generate Purely Negatively-Reinforced Text
Fabien Roger
When using adversarial training, it is common practice to train against the most egregious failures. However, this might imply using examples with sensitive information (such as le…
Language models are better than humans at next-token prediction
Buck Shlegeris, Fabien Roger, Lawrence Chan +1
Current language models are considered to have sub-human capabilities at natural language tasks like question-answering or writing code. However, language models are not trained to…