Focus on the Core: Efficient Attention via Pruned Token Compression for Document Classification

Abstract

Transformer-based models have achieved dominant performance in numerous NLP tasks. Despite their remarkable successes, pre-trained transformers such as BERT suffer from a computationally expensive self-attention mechanism that interacts with all tokens, including the ones unfavorable to classification performance. To overcome these challenges, we propose integrating two strategies: token pruning and token combining. Token pruning eliminates less important tokens in the attention mechanism's key and value as they pass through the layers. Additionally, we adopt fuzzy logic to handle uncertainty and alleviate potential mispruning risks arising from an imbalanced distribution of each token's importance. Token combining, on the other hand, condenses input sequences into smaller sizes in order to further compress the model. By integrating these two approaches, we not only improve the model's performance but also reduce its computational demands. Experiments with various datasets demonstrate superior performance compared to baseline models, especially with the best improvement over the existing BERT model, achieving +5%p in accuracy and +5.6%p in F1 score. Additionally, memory cost is reduced to 0.61x, and a speedup of 1.64x is achieved.

Accepted to EMNLP 2023 Findings

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Categorical Reparameterization with Gumbel-Softmax
• Longformer: The Long-Document Transformer
• Linformer: Self-Attention with Linear Complexity
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Object-Centric Learning with Slot Attention
• Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing
• Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations
• Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
• Token Merging: Your ViT But Faster
• SPViT: Enabling Faster Vision Transformers via Soft Token Pruning
• Learned Token Pruning for Transformers
• Less is More: Pay Less Attention in Vision Transformers
• Transformers are Short Text Classifiers: A Study of Inductive Short Text Classifiers on Benchmarks and Real-world Datasets
• DiffRate : Differentiable Compression Rate for Efficient Vision Transformers
• LittleBird: Efficient Faster & Longer Transformer for Question Answering