SALS: Sparse Attention in Latent Space for KV cache Compression

Junlin Mu, Hantao Huang, Jihang Zhang +3

Large Language Models capable of handling extended contexts are in high demand, yet their inference remains challenging due to substantial Key-Value cache size and high memory band…