EL-Attention: Memory Efficient Lossless Attention for Generation
arXiv:2105.04779
Abstract
Transformer model with multi-head attention requires caching intermediate results for efficient inference in generation tasks. However, cache brings new memory-related costs and prevents leveraging larger batch size for faster speed. We propose memory-efficient lossless attention (called EL-attention) to address this issue. It avoids heavy operations for building multi-head keys and values, cache for them is not needed. EL-attention constructs an ensemble of attention results by expanding query while keeping key and value shared. It produces the same result as multi-head attention with less GPU memory and faster inference speed. We conduct extensive experiments on Transformer, BART, and GPT-2 for summarization and question generation tasks. The results show EL-attention speeds up existing models by 1.6x to 5.3x without accuracy loss.
arXiv commentICML 2021. Version 2: add pseudocode
References in corpus (13)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- Longformer: The Long-Document Transformer
- Linformer: Self-Attention with Linear Complexity
- Reformer: The Efficient Transformer
- Big Bird: Transformers for Longer Sequences
- Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
- Pre-trained Summarization Distillation
- PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination
- Fast Transformer Decoding: One Write-Head is All You Need
- BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretraining
- Low-Rank Bottleneck in Multi-head Attention Models
- GLGE: A New General Language Generation Evaluation Benchmark