Hasse Diagrams for Attention: A Partial Order Framework for Designing Transformer Masks

Chentao Li, Han Guo

During the training of large Transformer models, attention masks regulate the scope and direction of information flow across a sequence. Numerous mask variants exist, and operators…