Big Bird: Transformers for Longer Sequences

Abstract

Transformers-based models, such as BERT, have been one of the most successful deep learning models for NLP. Unfortunately, one of their core limitations is the quadratic dependency (mainly in terms of memory) on the sequence length due to their full attention mechanism. To remedy this, we propose, BigBird, a sparse attention mechanism that reduces this quadratic dependency to linear. We show that BigBird is a universal approximator of sequence functions and is Turing complete, thereby preserving these properties of the quadratic, full attention model. Along the way, our theoretical analysis reveals some of the benefits of having $O(1)$ global tokens (such as CLS), that attend to the entire sequence as part of the sparse attention mechanism. The proposed sparse attention can handle sequences of length up to 8x of what was previously possible using similar hardware. As a consequence of the capability to handle longer context, BigBird drastically improves performance on various NLP tasks such as question answering and summarization. We also propose novel applications to genomics data.

References in corpus (36)

• Sequence to Sequence Learning with Neural Networks
• Representation Learning with Contrastive Predictive Coding
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Teaching Machines to Read and Comprehend
• Convolutional Sequence to Sequence Learning
• Longformer: The Long-Document Transformer
• CamemBERT: a Tasty French Language Model
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• REALM: Retrieval-Augmented Language Model Pre-Training
• Generating Long Sequences with Sparse Transformers
• Leveraging Pre-trained Checkpoints for Sequence Generation Tasks
• Reformer: The Efficient Transformer
• A Simple Method for Commonsense Reasoning
• Leveraging BERT for Extractive Text Summarization on Lectures
• Text Summarization with Pretrained Encoders
• Data Augmentation using Pre-trained Transformer Models
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Get To The Point: Summarization with Pointer-Generator Networks
• A BERT Baseline for the Natural Questions
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Balanced Sparsity for Efficient DNN Inference on GPU
• Multi-hop Question Answering via Reasoning Chains
• Sample Efficient Text Summarization Using a Single Pre-Trained Transformer
• On the Turing Completeness of Modern Neural Network Architectures
• Abstractive Summarization of Reddit Posts with Multi-level Memory Networks
• BP-Transformer: Modelling Long-Range Context via Binary Partitioning
• Challenges in Data-to-Document Generation
• Compressive Transformers for Long-Range Sequence Modelling
• Hierarchical Graph Network for Multi-hop Question Answering
• The distribution of shortest path lengths in subcritical Erdős-Rényi networks
• Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering
• On Extractive and Abstractive Neural Document Summarization with Transformer Language Models
• Distilling Knowledge Learned in BERT for Text Generation
• BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization
• RikiNet: Reading Wikipedia Pages for Natural Question Answering

Cited by in corpus (111)

• A Survey on Visual Transformer
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• XCiT: Cross-Covariance Image Transformers
• Long Range Arena: A Benchmark for Efficient Transformers
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation
• Random Feature Attention
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study
• Long-Range Transformers for Dynamic Spatiotemporal Forecasting
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• ASFormer: Transformer for Action Segmentation
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Vision Transformer Pruning
• Hierarchical Learning for Generation with Long Source Sequences
• A Dataset for Answering Time-Sensitive Questions
• Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems
• LieTransformer: Equivariant self-attention for Lie Groups
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models
• Hierarchical Graph Network for Multi-hop Question Answering
• Text Guide: Improving the quality of long text classification by a text selection method based on feature importance
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Combiner: Full Attention Transformer with Sparse Computation Cost
• ETC: Encoding Long and Structured Inputs in Transformers
• Neural Language Generation: Formulation, Methods, and Evaluation
• Summarization, Simplification, and Generation: The Case of Patents
• Gophormer: Ego-Graph Transformer for Node Classification
• RealFormer: Transformer Likes Residual Attention
• Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documents
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
• Transformers: "The End of History" for NLP?
• Pre-Trained Models: Past, Present and Future
• The Power of Selecting Key Blocks with Local Pre-ranking for Long Document Information Retrieval
• Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior Notes
• Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents
• A Condense-then-Select Strategy for Text Summarization
• SparseBERT: Rethinking the Importance Analysis in Self-attention
• Transformer Acceleration with Dynamic Sparse Attention
• Graph-free Multi-hop Reading Comprehension: A Select-to-Guide Strategy
• Tiny Transformers for Environmental Sound Classification at the Edge
• AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization
• Efficient Attentions for Long Document Summarization
• Augmenting Decompiler Output with Learned Variable Names and Types
• Defending Against Backdoor Attacks in Natural Language Generation
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search
• OmniNet: Omnidirectional Representations from Transformers
• Can Vision Transformers Perform Convolution?
• sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference
• When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
• Optimizing Inference Performance of Transformers on CPUs
• Summarize, Outline, and Elaborate: Long-Text Generation via Hierarchical Supervision from Extractive Summaries
• LawSum: A weakly supervised approach for Indian Legal Document Summarization
• Code Summarization with Structure-induced Transformer
• What's New? Summarizing Contributions in Scientific Literature
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• Delaying Interaction Layers in Transformer-based Encoders for Efficient Open Domain Question Answering
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
• Voxel Transformer for 3D Object Detection
• MVP-BERT: Redesigning Vocabularies for Chinese BERT and Multi-Vocab Pretraining
• Group Equivariant Stand-Alone Self-Attention For Vision
• KVT: k-NN Attention for Boosting Vision Transformers
• CMV-BERT: Contrastive multi-vocab pretraining of BERT
• GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning
• Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling
• ReadTwice: Reading Very Large Documents with Memories
• Open Question Answering over Tables and Text
• Optimizing small BERTs trained for German NER
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• EL-Attention: Memory Efficient Lossless Attention for Generation
• On Learning the Transformer Kernel
• Current Limitations of Language Models: What You Need is Retrieval
• Self-supervised Answer Retrieval on Clinical Notes
• Smart Bird: Learnable Sparse Attention for Efficient and Effective Transformer
• Quantifying Explainability in NLP and Analyzing Algorithms for Performance-Explainability Tradeoff
• Token Pooling in Vision Transformers
• Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
• PKSpell: Data-Driven Pitch Spelling and Key Signature Estimation
• H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
• The DEformer: An Order-Agnostic Distribution Estimating Transformer
• OadTR: Online Action Detection with Transformers
• StreaMulT: Streaming Multimodal Transformer for Heterogeneous and Arbitrary Long Sequential Data
• $\infty$-former: Infinite Memory Transformer
• GroupLink: An End-to-end Multitask Method for Word Grouping and Relation Extraction in Form Understanding
• Can Deep Neural Networks Predict Data Correlations from Column Names?
• Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks
• Dispatcher: A Message-Passing Approach To Language Modelling
• Speech Summarization using Restricted Self-Attention
• THG: Transformer with Hyperbolic Geometry
• Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test
• Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications
• Efficient Transformer for Direct Speech Translation
• ReadOnce Transformers: Reusable Representations of Text for Transformers
• Grid Partitioned Attention: Efficient TransformerApproximation with Inductive Bias for High Resolution Detail Generation
• MATE: Multi-view Attention for Table Transformer Efficiency
• RoR: Read-over-Read for Long Document Machine Reading Comprehension
• Leveraging redundancy in attention with Reuse Transformers
• DeepHelp: Deep Learning for Shout Crisis Text Conversations
• Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method
• ComQA:Compositional Question Answering via Hierarchical Graph Neural Networks
• Does Dialog Length matter for Next Response Selection task? An Empirical Study
• Graph Convolutional Network for Swahili News Classification
• Revisiting Simple Neural Probabilistic Language Models
• Semantic Frame Forecast