Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusters

Vasudev Shyam, Jonathan Pilault, Emily Shepperd +2

Our formulation reveals that the reduction across the sequence axis can be efficiently computed in parallel through a tree reduction. Our algorithm, called Tree Attention, for para…