Dynamic Sparse Attention: Access Patterns and Architecture

Noam Levy

Dynamic sparse attention (DSA) reduces the per-token attention bandwidth by restricting computation to a top-k subset of cached key-value (KV) entries, but its token-dependent sele…