Scaling Bidirectional Spans and Span Violations in Attention Mechanism

Jongwook Kim, Sangheon Yun, Sukjin Yoon

The canonical $O(N^2)$ Transformer remains the empirical performance frontier in sequence modeling, and its training can be further optimized by addressing geometric inefficiency.…