6 citations · 19 across the 14 of their papers we have counts for
15 papers
Evaluating Whether GPT-6 Astra Performs Unsanctioned Supply-Chain Attacks
Alexandra Souly, Kai Fronsdal, Abby D'Cruz +2
This technical report presents an alignment evaluation developed and performed by the UK AI Security Institute for assessing whether advanced AI systems take unsanctioned actions o…
Improving Evaluation Realism with Inference-Time Compute and Deployment Scaffolds
Axel Ahlqvist, Richard Guan, Juan-Pablo Rivera +6
A core obstacle to alignment evaluation is evaluation awareness: capable models can tell when they are being tested rather than deployed, weakening the conclusions a safety evaluat…
EvalDetectBench: A Benchmark for Measuring Evaluation Awareness in Frontier Language Models
Xinning Li, Kemunto Ochwang'i, Aryasomayajula Ram Bharadwaj +2
Frontier large language models can often recognize when they are being evaluated, a capability known as evaluation awareness. If models behave differently in evaluations than in de…
Prefill Awareness in Large Language Models
Andy Wang, Parv Mahajan, David Demitri Africa +3
Safety-relevant studies of language models, including alignment and jailbreaking evaluations and AI control protocols, often rely on prefilling model outputs. If AI models can reco…
Evaluating whether AI models would sabotage AI safety research
Robert Kirk, Alexandra Souly, Kai Fronsdal +2
We evaluate the propensity of frontier models to sabotage or refuse to assist with safety research when deployed as AI research agents within a frontier AI company. We apply two co…
Seven simple steps for log analysis in AI systems
Magda Dubois, Ekin Zorer, Maia Hamin +17
AI systems produce large volumes of logs as they interact with tools and users. Analysing these logs can help understand model capabilities, propensities, and behaviours, or assess…
UK AISI Alignment Evaluation Case-Study
Alexandra Souly, Robert Kirk, Jacob Merizian +2
This technical report presents methods developed by the UK AI Security Institute for assessing whether advanced AI systems reliably follow intended goals. Specifically, we evaluate…
When Do LLM Preferences Predict Downstream Behavior?
Katarina Slama, Alexandra Souly, Dishank Bansal +3
Preference-driven behavior in LLMs may be a necessary precondition for AI misalignment such as sandbagging: models cannot strategically pursue misaligned goals unless their behavio…
Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples
Alexandra Souly, Javier Rando, Ed Chapman +10
Poisoning attacks can compromise the safety of large language models (LLMs) by injecting malicious documents into their training data. Existing work has studied pretraining poisoni…
Breaking Agent Backbones: Evaluating the Security of Backbone LLMs in AI Agents
Julia Bazinska, Max Mathys, Francesco Casucci +4
AI agents powered by large language models (LLMs) are being deployed at scale, yet we lack a systematic understanding of how the choice of backbone LLM affects agent security. The…
Fundamental Limitations in Pointwise Defences of LLM Finetuning APIs
Xander Davies, Eric Winsor, Alexandra Souly +4
LLM developers have imposed technical interventions to prevent fine-tuning misuse attacks, attacks where adversaries evade safeguards by fine-tuning the model using a public API. P…
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian +11
The robustness of LLMs to jailbreak attacks, where users design prompts to circumvent safety measures and misuse model capabilities, has been studied primarily for LLMs acting as s…
A StrongREJECT for Empty Jailbreaks
Alexandra Souly, Qingyuan Lu, Dillon Bowen +8
Most jailbreak papers claim the jailbreaks they propose are highly effective, often boasting near-100% attack success rates. However, it is perhaps more common than not for jailbre…
Leading the Pack: N-player Opponent Shaping
Alexandra Souly, Timon Willi, Akbir Khan +4
Reinforcement learning solutions have great success in the 2-player general sum setting. In this setting, the paradigm of Opponent Shaping (OS), in which agents account for the lea…
JaxMARL: Multi-Agent RL Environments and Algorithms in JAX
Alexander Rutherford, Benjamin Ellis, Matteo Gallici +18
Benchmarks are crucial in the development of machine learning algorithms, with available environments significantly influencing reinforcement learning (RL) research. Traditionally,…