Extra Global Attention Designation Using Keyword Detection in Sparse Transformer Architectures

Evan Lucas, Dylan Kangas, Timothy C Havens

In this paper, we propose an extension to Longformer Encoder-Decoder, a popular sparse transformer architecture. One common challenge with sparse transformers is that they can stru…