Rethinking Heterogeneous System Disaggregation for Subquadratic Attention

Arya Tschand, Yaosheng Fu, Vikram Sharma Mailthody +6

Frontier language models are more aggressively using subquadratic attention to reduce the memory footprint and compute requirements during inference while still delivering frontier…