Attention over pre-trained Sentence Embeddings for Long Document Classification

Amine Abdaoui, Sourav Dutta

Despite being the current de-facto models in most NLP tasks, transformers are often limited to short sequences due to their quadratic attention complexity on the number of tokens.…