32 papers
Functional Welfare for AI Safety
Rikhil Jhaveri, Jamie Johnson, Punya Syon Pandey +1
AI welfare is generally treated purely as an ethical issue rather than also as a safety issue. We argue this is a mistake. Whatever their moral status, models exhibit welfare-like…
Character Training for Risk-Averse Agents
Arav Dhoot, Punya Syon Pandey, Jamie Johnson +3
Risk aversion in resources could prevent misaligned AI agents from causing catastrophic harm. Misaligned but risk-averse agents would tend to favor safer strategies like making dea…
Stress-testing Alignment Midtraining
Sid Baines, Jonathan Bostock, Maria Angelica Martinez +3
When aligning frontier models through post-training techniques, it is not possible to directly demonstrate all of the behaviours we want a model to exhibit in all possible deployme…
Inoculation Midtraining with Learned Neologisms
Kyle O'Brien, Edward James Young, Puria Radmard +4
Large language models (LLMs) often learn both desirable and undesirable properties during post-training. We study whether midtraining, an earlier training stage, can shape which of…
Last Translation Benchmark
Vilém Zouhar, Niyati Bafna, Mukund Choudhary +263
For scientific progress, we need benchmarks that test the limits of state-of-the-art models, and evaluation methods that inform us about failure cases. As models get stronger, stan…
Measuring Activation Control in Large Language Models
Marek Mateusz Kowalski, Joshua Fonseca Rivera, Uzay Macar +1
Safe deployment of increasingly capable models will likely come to rely on latent-space monitoring as a complement to behavioral evaluations, especially when evaluation-aware model…
Can AI agents conduct open-ended AI research? Early evidence from two case studies
Peter Kirgis, Sayash Kapoor, Andrew Schwartz +21
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations eithe…
Item Response Theory for AI Safety
Joshua Fonseca Rivera, Neil Shah, David Demitri Africa +1
Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to trust and interpret, because b…
Detecting CSAM Text-to-Image LoRAs From Weights
David Demitri Africa, Cate Heine, Nadine Staes-Polet +1
Low-rank adaptation (LoRA) fine-tuning has made it cheap and easy to customize open-weight image generation models for specific tasks, including the production of child sexual abus…
Persona Cartography: Charting Language Model Personality Traits in Weight Space
Luke Baines, Anton Gonzalvez Hawthorne, Mariia Koroliuk +4
Large language models exhibit recurring behavioural patterns -- personas -- that shape generalisation and safety, but we lack reliable tools for decomposing, measuring, and control…
Tool Use Enables Undetectable Steganography in Multi-Agent LLM Systems
Jimmy Laurence Rippin, Simon C. Marshall, David Demitri Africa +1
Increasingly autonomous agentic AI systems pose novel multi-agent risks, such as secret collusion via covert communication channels. The natural defence to these collusion attempts…
When Role-playing, Do Models Believe What They Say?
Benjamin Sturgeon, David Africa, Sid Black
Language models can state that "the Earth orbits the Sun" and, when role-playing Aristotle, assert the opposite. Recent work argues that persona adoption is fundamental to how lang…
"Did you lie?" Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms
Alan Cooney, David Africa, Geoffrey Irving
Robust lie detectors for language models could enable powerful techniques for auditing, monitoring, and post-hoc investigation of model behaviour, but evaluating them requires test…
PACUTE: Phonology-, Affix-, and Character-level Understanding of Tokens for Filipino
Jann Railey Montalan, David Demitri Africa, Jimson Paulo Layacan +3
Large language models (LLMs) process text as sequences of subword tokens, which can obscure the character-level and morphological structure that underlies word formation. This limi…
Prefill Awareness in Large Language Models
Andy Wang, Parv Mahajan, David Demitri Africa +3
Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can reco…
Consistency Training Along the Transformer Stack
Sukrati Gautam, Neil Shah, Arav Dhoot +7
Consistency training encourages models to behave similarly across different contexts, and has shown promise for reducing misalignment. We broaden the scope of consistency training…
Consistency Training Can Entrench Misalignment
David Demitri Africa, Arathi Mani
Consistency training encourages a model to produce similar outputs across related inputs or sampling procedures. Such methods are simple, scalable, and largely label-free, but thei…
Consistency Training while Mitigating Obfuscation via Rate Matching
Sohaib Imran, Prakhar Gupta, Jannes Elstner +1
Large language models are often influenced by extraneous input features, such as cues revealing a user's preferred answer. Consistency training reduces this influence by training m…
LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
Igor Ivanov, David Demitri Africa
Large language models can recognize when they are being evaluated (evaluation awareness) and behave differently because of that, which undermines the validity of safety and alignme…
Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
Cameron Tice, Puria Radmard, Samuel Ratnam +3
Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood. If prevailing descri…
A Decision-Theoretic Formalisation of Steganography With Applications to LLM Monitoring
Usman Anwar, Julianna Piskorz, David D. Baek +6
Large language models are beginning to show steganographic capabilities. Such capabilities could allow misaligned models to evade oversight mechanisms. Yet principled methods to de…
Does Self-Evaluation Enable Wireheading in Language Models?
David Demitri Africa, Hans Ethan Ting
Self-evaluation is increasingly central to language model training, underpinning techniques from Constitutional AI to self-refinement. We investigate whether coupling self-evaluati…
Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
Daniel Tan, Anders Woodruff, Niels Warncke +4
Language model finetuning often results in learning undesirable traits in combination with desired ones. To address this, we propose inoculation prompting: modifying finetuning dat…
Steering Awareness: Detecting Activation Steering from Within
Joshua Fonseca Rivera, David Demitri Africa
Activation steering -- adding a vector to a model's residual stream to modify its behavior -- is widely used in safety evaluations as if the model cannot detect the intervention. W…