24 citations · 35 across the 9 of their papers we have counts for
6 papers · 1 filter
Transformers Can Navigate Mazes With Multi-Step Prediction
Niklas Nolte, Ouail Kitouni, Adina Williams +2
Despite their remarkable success in language modeling, transformers trained to predict the next token in a sequence struggle with long-term planning. This limitation is particularl…
The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
Ouail Kitouni, Niklas Nolte, Diane Bouchacourt +3
Today's best language models still struggle with hallucinations: factually incorrect generations, which impede their ability to reliably retrieve information seen during training.…
From Neurons to Neutrons: A Case Study in Interpretability
Ouail Kitouni, Niklas Nolte, Víctor Samuel Pérez-Díaz +2
Mechanistic Interpretability (MI) promises a path toward fully understanding how neural networks make their predictions. Prior work demonstrates that even when trained to perform s…
DiSK: A Diffusion Model for Structured Knowledge
Ouail Kitouni, Niklas Nolte, James Hensman +1
Structured (dictionary-like) data presents challenges for left-to-right language models, as they can struggle with structured entities for a wide variety of reasons such as formatt…
Expressive Monotonic Neural Networks
Ouail Kitouni, Niklas Nolte, Michael Williams
The monotonic dependence of the outputs of a neural network on some of its inputs is a crucial inductive bias in many scenarios where domain knowledge dictates such behavior. This…
Towards Understanding Grokking: An Effective Theory of Representation Learning
Ziming Liu, Ouail Kitouni, Niklas Nolte +3
We aim to understand grokking, a phenomenon where models generalize long after overfitting their training set. We present both a microscopic analysis anchored by an effective theor…