LSG Attention: Extrapolation of pretrained Transformers to long sequences

Charles Condevaux, Sébastien Harispe

Transformer models achieve state-of-the-art performance on a wide range of NLP tasks. They however suffer from a prohibitive limitation due to the self-attention mechanism, inducin…