Taming the Fragility of KV Cache Eviction in LLM Inference

Yuan Feng, Haoyu Guo, JunLin Lv +2

Large language models have revolutionized natural language processing, yet their deployment remains hampered by the substantial memory and runtime overhead of the transformer's Key…