39 citations · 88 across the 63 of their papers we have counts for
24 papers · 1 filter
Causal Chain of Thought: Mitigating Attention Leakage via Context Isolation
Gabriele Dominici, Teague McMillan, Fazl Barez +2
Chain-of-thought (CoT) prompting elicits step-by-step rationales that appear to explain model predictions, but there is no guarantee these steps causally drive the final answer. In…
Pretraining Curricula Enable Selective Fine-tuning
Sebastian A. Bruijns, Jirko Rubruck, Mia H. Whitefield +3
Transformers follow implicit curricula whereby some tasks are learned before others. However, how explicit pretraining curricula influence learning, generalization, and the selecti…
Running the Gauntlet: Hard Agentic Tasks
Mykola Vysotskyi, Runqi Lin, Grzegorz Biziel +21
As agentic systems continue to evolve and are widely deployed in real-world scenarios, there is a growing demand to faithfully evaluate their capabilities. However, current benchma…
Interpretability Can Be Actionable
Hadas Orgad, Fazl Barez, Tal Haklay +9
Interpretability aims to explain the behavior of deep neural networks. Despite rapid growth, there is mounting concern that much of this work has not translated into practical impa…
Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value
Joe Edelman, Tan Zhi-Xuan, Ryan Lowe +30
Beneficial societal outcomes cannot be guaranteed by aligning individual AI systems with the intentions of their operators or users. Even an AI system that is perfectly aligned to…
Beyond Linear Probes: Dynamic Safety Monitoring for Language Models
James Oldfield, Philip Torr, Ioannis Patras +2
Monitoring large language models' (LLMs) activations is an effective way to detect harmful requests before they lead to unsafe outputs. However, traditional safety monitors often r…
Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer
Simon Schrodi, Elias Kempf, Fazl Barez +1
Language models can transfer hidden biases during distillation. For example, a teacher that "likes owls" can make its student "like owls" too, even when the training data consists…
Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
Minseon Kim, Jin Myung Kwak, Lama Alssum +5
Fine-tuning language models is commonly believed to inevitably harm their safety, i.e., refusing to respond to harmful user requests, even when using harmless datasets, thus requir…
Scaling sparse feature circuit finding for in-context learning
Dmitrii Kharlapenko, Stepan Shabalin, Fazl Barez +2
Sparse autoencoders (SAEs) are a popular tool for interpreting large language model activations, but their utility in addressing open questions in interpretability remains unclear.…
Do Sparse Autoencoders Generalize? A Case Study of Answerability
Lovis Heindrich, Philip Torr, Fazl Barez +1
Sparse autoencoders (SAEs) have emerged as a promising approach in language model interpretability, offering unsupervised extraction of sparse features. For interpretability method…
Open Problems in Machine Unlearning for AI Safety
Fazl Barez, Tingchen Fu, Ameya Prabhu +16
As AI systems become more capable, widely deployed, and increasingly autonomous in critical areas such as cybersecurity, biological research, and healthcare, ensuring their safety…
Jailbreak Defense in a Narrow Domain: Limitations of Existing Methods and a New Transcript-Classifier Approach
Tony T. Wang, John Hughes, Henry Sleight +7
Defending large language models against jailbreaks so that they never engage in a broadly-defined set of forbidden behaviors is an open problem. In this paper, we investigate the d…
Enhancing Neural Network Interpretability with Feature-Aligned Sparse Autoencoders
Luke Marks, Alasdair Paren, David Krueger +1
Sparse Autoencoders (SAEs) have shown promise in improving the interpretability of neural network activations, but can learn features that are not features of the input, limiting t…
Quantifying Feature Space Universality Across Large Language Models via Sparse Autoencoders
Michael Lan, Philip Torr, Austin Meek +3
The Universality Hypothesis in large language models (LLMs) claims that different models converge towards similar concept representations in their latent spaces. Providing evidence…
Risks and Opportunities of Open-Source Generative AI
Francisco Eiras, Aleksandar Petrov, Bertie Vidgen +22
Applications of Generative AI (Gen AI) are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for these seismic chan…
Near to Mid-term Risks and Opportunities of Open-Source Generative AI
Francisco Eiras, Aleksandar Petrov, Bertie Vidgen +21
In the next few years, applications of Generative AI are expected to revolutionize a number of different areas, ranging from science & medicine to education. The potential for thes…
Visualizing Neural Network Imagination
Nevan Wichers, Victor Tao, Riccardo Volpato +1
In certain situations, neural networks will represent environment states in their hidden activations. Our goal is to visualize what environment states the networks are representing…
Understanding Addition and Subtraction in Transformers
Philip Quirke, Clement Neo, Fazl Barez
We use integer addition and subtraction as a controlled, exactly-solvable testbed for what can be said with confidence about the algorithm a low-loss transformer implements - logic…
DeepDecipher: Accessing and Investigating Neuron Activation in Large Language Models
Albert Garde, Esben Kran, Fazl Barez
As large language models (LLMs) become more capable, there is an urgent need for interpretable and transparent tools. Current methods are difficult to implement, and accessible too…
Understanding Addition in Transformers
Philip Quirke, Fazl Barez
Understanding the inner workings of machine learning models like Transformers is vital for their safe and ethical use. This paper provides a comprehensive analysis of a one-layer T…
Interpreting Learned Feedback Patterns in Large Language Models
Luke Marks, Amir Abdullah, Clement Neo +4
Reinforcement learning from human feedback (RLHF) is widely used to train large language models (LLMs). However, it is unclear whether LLMs accurately learn the underlying preferen…
Neuron to Graph: Interpreting Language Model Neurons at Scale
Alex Foote, Neel Nanda, Esben Kran +3
Advances in Large Language Models (LLMs) have led to remarkable capabilities, yet their inner mechanisms remain largely unknown. To understand these models, we need to unravel the…
System III: Learning with Domain Knowledge for Safety Constraints
Fazl Barez, Hosien Hasanbieg, Alesandro Abbate
Reinforcement learning agents naturally learn from extensive exploration. Exploration is costly and can be unsafe in domains. This paper proposes a novel…
N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models
Alex Foote, Neel Nanda, Esben Kran +2
Understanding the function of individual neurons within language models is essential for mechanistic interpretability research. We propose , a tool…