RoVE: Rotary Value Embeddings Attention for Relative Position-dependent Value Pathways

Alejandro García-Castellanos, Maurice Weiler, Erik J Bekkers

Rotary Position Embeddings (RoPE) make attention scores position-relative but leave the value pathway position-blind: the message sent by a value token is the same regardless of it…