AttnFuse: A Composable DSL for Compiling Attentions to Fused GPU Kernels

Varun Kumar Dasoju, Tian Zhao

Modern AI systems are built on the Transformer architecture, whose core operation, attention, accounts for the majority of computation and memory cost. Researchers continually prop…