Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
arXiv:2108.13032
Abstract
The highly popular Transformer architecture, based on self-attention, is the foundation of large pretrained models such as BERT, that have become an enduring paradigm in NLP. While powerful, the computational resources and time required to pretrain such models can be prohibitive. In this work, we present an alternative self-attention architecture, Shatter, that more efficiently encodes sequence information by softly partitioning the space of relative positions and applying different value matrices to different parts of the sequence. This mechanism further allows us to simplify the multi-headed attention in Transformer to single-headed. We conduct extensive experiments showing that Shatter achieves better performance than BERT, with pretraining being faster per step (15% on TPU), converging in fewer steps, and offering considerable memory savings (>50%). Put together, Shatter can be pretrained on 8 V100 GPUs in 7 days, and match the performance of BERT_Base -- making the cost of pretraining much more affordable.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Longformer: The Long-Document Transformer
- SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Generating Long Sequences with Sparse Transformers
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- Reformer: The Efficient Transformer
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Rethinking Positional Encoding in Language Pre-training
- Are Sixteen Heads Really Better than One?
- EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets
- The MultiBERTs: BERT Reproductions for Robustness Analysis