Content-Based Addressing for Long Context

Mahesh Godavarti

Rotary position embedding (RoPE) uses each token's integer position to determine the rotation applied inside attention. This works well for local token order, but increasing contex…