RoPE attention is an exact forward-pass gradient step with softmax intact

Julie Huang, Maggie Chlon, Leon Chlon

We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we c…