RATTENTION: Towards the Minimal Sliding Window Size in Local-Global Attention Models

Bailin Wang, Chang Lan, Chong Wang +1

Local-global attention models have recently emerged as compelling alternatives to standard Transformers, promising improvements in both training and inference efficiency. However,…