S2-Attention: Hardware-Aware Context Sharding Among Attention Heads

Xihui Lin, Yunan Zhang, Suyu Ge +5

Sparse attention, which selectively attends to a subset of tokens in the context was supposed to be efficient. However, its theoretical reduction in FLOPs has rarely translated int…