FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention

Guoxia Wang, Qianyue He, Siming Wu +6

Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node…