Shifting Mechanisms: How Positional Encoding Choice Shapes In-Context Retrieval

Eric Enouen, Sainyam Galhotra

Language models increasingly use architectures that vary attention span and positional encoding across layers, such as applying RoPE with sliding-window attention and NoPE with glo…