ScaleFormer: Span Representation Cumulation for Long-Context Transformer

Jiangshu Du, Wenpeng Yin, Philip Yu

The quadratic complexity of standard self-attention severely limits the application of Transformer-based models to long-context tasks. While efficient Transformer variants exist, t…