NOSA: Native and Offloadable Sparse Attention

Yuxiang Huang, Pengjie Wang, Jicheng Han +9

Decoding throughput improvements from larger inference batches are limited by GPU memory, which is largely consumed by the key-value (KV) cache. Prior training-free KV cache offloa…