CommunityKV: Efficient Long-Context Decoding via Graph Partitioning

Joe McKenna, Anastasios Alexandridis, Nathan Susanj +1

Scaling Transformers to long contexts is constrained by the quadratic cost of self-attention and the linear growth of key-value cache memory transfer. Sparse attention mitigates th…