Separators Make Carry Propagation Learnable:The Geometry of Latent Carry in a Multiplication Transformer

Sama Satariyan, Raphael Cousin, G{é}rard Biau

Transformers asked to multiply multi-digit numbers in a single forward pass often fail, and interpretability studies of pretrained language models find arithmetic solved by input-r…