Autonomy-of-Heads: Data-Free Sparse Attention from Frozen Query-Key Geometry

Yehan Yang, Junyuan Shang, Yang Li +3

Long-context LLM inference is bottlenecked by quadratic attention computation and growing KV-cache costs. Existing sparse attention and KV-compression methods typically decide whic…