Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Jungmin Yun, Mihyeon Kim, Youngbin Kim

Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computati…