Simple Local Attentions Remain Competitive for Long-Context Tasks

Wenhan Xiong, Barlas Oğuz, Anchit Gupta +5

Many NLP tasks require processing long contexts beyond the length limit of pretrained models. In order to scale these models to longer text sequences, many efficient long-range att…