Fovea Transformer: Efficient Long-Context Modeling with Structured Fine-to-Coarse Attention

Ziwei He, Jian Yuan, Le Zhou +2

The quadratic complexity of self-attention in Transformers has hindered the processing of long text. To alleviate this problem, previous works have proposed to sparsify the attenti…