LazyFormer: Self Attention with Lazy Update
arXiv:2102.12702
Abstract
Improving the efficiency of Transformer-based language pre-training is an important task in NLP, especially for the self-attention module, which is computationally expensive. In this paper, we propose a simple but effective solution, called \emph{LazyFormer}, which computes the self-attention distribution infrequently. LazyFormer composes of multiple lazy blocks, each of which contains multiple Transformer layers. In each lazy block, the self-attention distribution is only computed once in the first layer and then is reused in all upper layers. In this way, the cost of computation could be largely saved. We also provide several training tricks for LazyFormer. Extensive experiments demonstrate the effectiveness of the proposed method.
References in corpus (8)
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Longformer: The Long-Document Transformer
- Mixed Precision Training
- Linformer: Self-Attention with Linear Complexity
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- Rethinking Attention with Performers
- Sharing Attention Weights for Fast Transformer