Selective Induction Heads: How Transformers Select Causal Structures In Context

Francesco D'Angelo, Francesco Croce, Nicolas Flammarion

Transformers have exhibited exceptional capabilities in sequence modeling tasks, leveraging self-attention and in-context learning. Critical to this success are induction heads, at…