Where Activation Sparsity and KV-Cache Sparsity Cross in LLM Decoding

Jungseob Lee, Seungyoon Lee, Seongtae Hong +2

At each step, decoding one sequence with a large language model rereads the projection weights, whose traffic is fixed, and the key-value (KV) cache, whose traffic grows with conte…