1 citations · 1 across the 2 of their papers we have counts for
5 papers
Model Organisms of Collusion: Corrective Transfer in Untrusted Monitors
Juan-Pablo Rivera, Ari Spiesberger, Juan J Vazquez +6
AI control protocols aim to get useful work from models that may be misaligned by monitoring their behaviour. One approach uses a second copy of the same model as the monitor. This…
Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Arun Jose, Julian Stastny
Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (…
Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
Daniel Tan, Anders Woodruff, Niels Warncke +4
Language model finetuning often results in learning undesirable traits in combination with desired ones. To address this, we propose inoculation prompting: modifying finetuning dat…
Reasoning Models Sometimes Output Illegible Chains of Thought
Arun Jose
Language models trained via outcome-based reinforcement learning (RL) to reason using chain-of-thought (CoT) have shown remarkable performance. Monitoring such a model's CoT may al…
Why Do Some Language Models Fake Alignment While Others Don't?
Abhay Sheshadri, John Hughes, Julian Michael +4
Alignment faking in large language models presented a demonstration of Claude 3 Opus and Claude 3.5 Sonnet selectively complying with a helpful-only training objective to prevent m…