H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
arXiv:2107.11906
Abstract
We describe an efficient hierarchical method to compute attention in the Transformer architecture. The proposed attention mechanism exploits a matrix structure similar to the Hierarchical Matrix (H-Matrix) developed by the numerical analysis community, and has linear run time and memory complexity. We perform extensive experiments to show that the inductive bias embodied by our hierarchical attention is effective in capturing the hierarchical structure in the sequences typical for natural language and vision tasks. Our method is superior to alternative sub-quadratic proposals by over +6 points on average on the Long Range Arena benchmark. It also sets a new SOTA test perplexity on One-Billion Word dataset with 5x fewer model parameters than that of the previous-best Transformer-based models.
arXiv commentACL2021 long paper oral presentation
References in corpus (12)
- Language Models are Few-Shot Learners
- Longformer: The Long-Document Transformer
- Linformer: Self-Attention with Linear Complexity
- Effective Approaches to Attention-based Neural Machine Translation
- Generating Long Sequences with Sparse Transformers
- Axial Attention in Multidimensional Transformers
- Reformer: The Efficient Transformer
- Big Bird: Transformers for Longer Sequences
- Stand-Alone Self-Attention in Vision Models
- Synthesizer: Rethinking Self-Attention in Transformer Models
- Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
- Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers