8 citations · 9 across the 5 of their papers we have counts for
5 papers
Measuring Reward-Seeking via Contrastive Belief Updates
Axel Højmark, Jérémy Scheurer, Evgenia Nitishinskaya +5
Language models trained with reinforcement learning may learn to optimize the grader's judgment rather than the intended objective. This "reward-seeking" is difficult to measure be…
Stress Testing Deliberative Alignment for Anti-Scheming Training
Bronson Schoen, Evgenia Nitishinskaya, Mikita Balesni +16
Highly capable AI systems could secretly pursue misaligned goals -- what we call "scheming". Because a scheming AI would deliberately try to hide its misaligned goals and actions,…
Trading Inference-Time Compute for Adversarial Robustness
Wojciech Zaremba, Evgenia Nitishinskaya, Boaz Barak +8
We conduct experiments on the impact of increasing inference-time compute in reasoning models (specifically OpenAI o1-preview and o1-mini) on their robustness to adversarial attack…
LLM Critics Help Catch LLM Bugs
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe +3
Reinforcement learning from human feedback (RLHF) is fundamentally limited by the capacity of humans to correctly evaluate model output. To improve human evaluation ability and ove…
Generalized Wick Decompositions
Chris MacLeod, Evgenia Nitishinskaya, Buck Shlegeris
We review the cumulant decomposition (a way of decomposing the expectation of a product of random variables (e.g. ) into a sum of terms corresponding to partitions…