What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents

Ashwin Gerard Colaco, Nada Lahjouji

Large language models, and the agents built on them, spend an ever-growing share of their compute and memory on remembering: caching attention keys and values, carrying long prompt…