3 papers
cs.LG2025
Exact Learning of Arithmetic with Differentiable Agents
Hristo Papazov, Francesco D'Angelo, Nicolas Flammarion
We explore the possibility of exact algorithmic learning with gradient-based methods and introduce a differentiable framework capable of strong length generalization on arithmetic…
cs.LG2025
Selective Induction Heads: How Transformers Select Causal Structures In Context
Francesco D'Angelo, Francesco Croce, Nicolas Flammarion
Transformers have exhibited exceptional capabilities in sequence modeling tasks, leveraging self-attention and in-context learning. Critical to this success are induction heads, at…
cs.LG2025
The emergence of sparse attention: impact of data distribution and benefits of repetition
Nicolas Zucchet, Francesco d'Angelo, Andrew K. Lampinen +1
Emergence is a fascinating property of large language models and neural networks more broadly: as models scale and train for longer, they sometimes develop new abilities in sudden…