Written as a Record, Read as an Address: What a Forward Pass Leaves in an Operation's KV Cache

Lingfeng Wu, Behzad Shomali

When a language model reads an operation such as "Swap the contents of Box F and Box B", its forward pass writes keys and values for those tokens into the KV cache. Prior work on e…