paper

Systolic Linear Attention: Co-Designing Tile-Grouped Gated Delta Attention for TPUs

arXiv:2611.06828

Abstract

Gated-delta linear attention layers such as Gated DeltaNet (GDN), Kimi Delta Attention (KDA), and GDN-2 are tuned for GPUs, where small tensor-core tiles and fast vector registers make unpacked heads, per-channel decay rescaling, and per-value-head chunk solves cheap. On Google TPUs, which concentrate compute in wide systolic matrix units backed by a slower vector unit, direct ports leave half of each matrix tile padded (for heads) and repeat decay rescaling and chunk solves across every value head. We present \textbf{Systolic Linear Attention (SLA)}, which reshapes gated delta attention around the matrix unit by widening keys to and sharing both keys and scalar write gates across grouped value heads ( fewer chunk solves), grouping decay gates into four -channel tiles, and fusing exact FP32 block-doubling chunk inversion and its backward reuse inside one on-chip kernel. With exact chunk inversion on TPU v7x, -stage \texttt{SLA-R4} (four value heads per key head) runs -- faster than open-source MaxText KDA and -- faster than open-source Tokamax KDA, while using -- less temporary high-bandwidth memory (HBM). Across a -chip TPU v7x pre-training study of sparse Mixture-of-Experts (MoE) models from to nominal parameters, SLA trains -- faster than MaxText KDA (-- faster than Tokamax KDA), matches KDA perplexity on three paired seeds at , and stays within at other scales.