FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention

Anthony Givans, Michael Crawshaw, Mingrui Liu

Transformer models built on the attention mechanism have become a central building block in modern deep learning, yet softmax attention remains a major bottleneck for long-context…