The I/O Complexity of Attention, or How Optimal is Flash Attention?

Barna Saha, Christopher Ye

Self-attention is at the heart of the popular Transformer architecture, yet suffers from quadratic time and memory complexity. The breakthrough FlashAttention algorithm revealed I/…