Blockwise Self-Attention for Long Document Understanding
arXiv:1911.02972
Abstract
We present BlockBERT, a lightweight and efficient BERT model for better modeling long-distance dependencies. Our model extends BERT by introducing sparse block structures into the attention matrix to reduce both memory consumption and training/inference time, which also enables attention heads to capture either short- or long-range contextual information. We conduct experiments on language model pre-training and several benchmark question answering datasets with various paragraph lengths. BlockBERT uses 18.7-36.1% less memory and 12.0-25.1% less time to learn the model. During testing, BlockBERT saves 27.8% inference time, while having comparable and sometimes better prediction accuracy, compared to an advanced BERT-based model, RoBERTa.
arXiv commentAccepted at Findings of EMNLP'20 and SustaiNLP 2020 at EMNLP'20, 12 pages
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- Language Models are Few-Shot Learners
- Longformer: The Long-Document Transformer
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
- Mixed Precision Training
- Training Deep Nets with Sublinear Memory Cost
- Generating Long Sequences with Sparse Transformers
- SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
- Passage Re-ranking with BERT
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- fairseq: A Fast, Extensible Toolkit for Sequence Modeling
- CondenseNet: An Efficient DenseNet using Learned Group Convolutions
- Evaluating Protein Transfer Learning with TAPE
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
- Are Sixteen Heads Really Better than One?
- Low-Memory Neural Network Training: A Technical Report
- Adaptively Sparse Transformers
Cited by in corpus (5)
- Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
- Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
- Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
- Long-Short Transformer: Efficient Transformers for Language and Vision
- T-EMDE: Sketching-based global similarity for cross-modal retrieval