History Replays Itself: Unleashing Long-Context Capability in Linear Attention

Li Nanbo, Firas Laakom, Yilan Zhang +10

Linear attention promises the efficiency needed for long-context language modelling, yet its fixed-size recurrent state must simultaneously support ongoing computation and preserve…