6 papers
Fast and Geometrically Grounded Lorentz Neural Networks
Robert van der Klis, Ricardo Chávez Torres, Max van Spengler +3
Hyperbolic space is quickly gaining traction as a promising geometry for hierarchical and robust representation learning. A core open challenge is the development of a mathematical…
On the Emergence of Induction Heads for In-Context Learning
Tiberiu Musat, Tiago Pimentel, Lorenzo Noci +3
Transformers have become the dominant architecture for natural language processing. Part of their success is owed to a remarkable capability known as in-context learning (ICL): the…
Planner and Executor: Collaboration between Discrete Diffusion And Autoregressive Models in Reasoning
Lina Berrayana, Ahmed Heakl, Muhammad Abdullah Sohail +3
Current autoregressive language models (ARMs) achieve high accuracy but require long token sequences, making them costly. Discrete diffusion language models (DDLMs) enable parallel…
The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?
Denis Sutter, Julian Minder, Thomas Hofmann +1
The concept of causal abstraction got recently popularised to demystify the opaque decision-making processes of machine learning models; in short, a neural network can be abstracte…
Causal Estimation of Tokenisation Bias
Pietro Lesci, Clara Meister, Thomas Hofmann +2
Modern language models are typically trained over subword sequences, but ultimately define probabilities over character-strings. Ideally, the choice of the tokeniser -- which maps…
Optimality of Right-Invariant Priors
Jannis Bolik, Thomas Hofmann
We discuss optimal prediction for families of probability distributions with a locally compact topological group structure. Right-invariant priors were previously shown to yield a…