Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers

Jonathan Lys, Vincent Gripon, Bastien Pasdeloup +4

Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignme…