2 citations · 2 across the 5 of their papers we have counts for
6 papers
Audit tools of the trade: Automated Auditor Elicitation Can Be Sensitive to the Context the Auditor Is Given
Ben Slater, Ryan Lundqvist, Leo Richter +1
Does providing additional context to automated AI auditors improve their effectiveness? Using the Petri tool to audit LLM coding agents, we test the impact of providing two types o…
A False Average: Pooled CoT-Monitor Accuracy Conceals a Reasoning-Dependent Fragility
Shikhar Shiromani, Leo Richter, Noah Siegel
Chain-of-thought (CoT) monitors are reported by their aggregate accuracy on a pool of reward hacks. We show that this number is a false average. On Terminal Wrench, about 77% of ha…
Plausible Deniability Guarantees for Whistleblowers
Leo Richter, Matt J. Kusner
Whistleblowers are a key safeguard against organizational wrongdoing, but the threat of retaliation deters reporting. Existing whistleblower-protection proposals lack formal privac…
Agentic Uncertainty Reveals Agentic Overconfidence
Jean Kaddour, Srijan Patel, Gbètondji Dovonon +3
Can AI agents predict whether they will succeed at a task? We study agentic uncertainty by eliciting success probability estimates before, during, and after task execution. All res…
ContextBench: Modifying Contexts for Targeted Latent Activation
Robert Graham, Edward Stevinson, Leo Richter +3
Identifying inputs that trigger specific behaviours or latent features in language models could have a wide range of safety use cases. We investigate a class of methods capable of…
An Auditing Test To Detect Behavioral Shift in Language Models
Leo Richter, Xuanli He, Pasquale Minervini +1
As language models (LMs) approach human-level performance, a comprehensive understanding of their behavior becomes crucial. This includes evaluating capabilities, biases, task perf…