A Hippocampus for Linear Attention: An Exact Memory for What the Recurrent State Forgets

Wanyun Cui

Linear-attention and state-space language models compress the prefix into a fixed-size recurrent state, yielding O(1) memory at the cost of a lossy exact memory: when many key--val…