13 citations · 18 across the 15 of their papers we have counts for
5 papers · 1 filter
Heads collapse, features stay: Why Replay needs big buffers
Giulia Lanzillotta, Damiano Meier, Thomas Hofmann
A persistent paradox in continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their output predictions fail. We fo…
On the Emergence of Induction Heads for In-Context Learning
Tiberiu Musat, Tiago Pimentel, Lorenzo Noci +3
Transformers have become the dominant architecture for natural language processing. Part of their success is owed to a remarkable capability known as in-context learning (ICL): the…
Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
Lina Berrayana, Ahmed Heakl, Muhammad Abdullah Sohail +3
Current autoregressive language models (ARMs) achieve high accuracy but require long token sequences, making them costly. Discrete diffusion language models (DDLMs) enable parallel…
The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Denis Sutter, Julian Minder, Thomas Hofmann +1
The concept of causal abstraction got recently popularised to demystify the opaque decision-making processes of machine learning models; in short, a neural network can be abstracte…
Causal Estimation of Tokenisation Bias
Pietro Lesci, Clara Meister, Thomas Hofmann +2
Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser -- which maps…