Efficient Long-Range Transformers: You Need to Attend More, but Not Necessarily at Every Layer

Qingru Zhang, Dhananjay Ram, Cole Hawkins +2

Pretrained transformer models have demonstrated remarkable performance across various natural language processing tasks. These models leverage the attention mechanism to capture lo…