CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective

Yuan Feng, Junlin Lv, Haoyu Guo +3

Large language models have revolutionized natural language processing but face significant challenges of high storage and runtime costs, due to the transformer architecture's relia…