Shallow Encoding Emits Deep Memories

Mingyuan Ma, Yangmin Li, Harris Nover +4

Hybrid language models interleave a few softmax-attention layers with many linear-attention or state-space layers, so every layer keeps a memory of earlier tokens: a key-value cach…