Reformer: The Efficient Transformer

Abstract

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensitive hashing, changing its complexity from O($L^2$) to O($L\log L$), where $L$ is the length of the sequence. Furthermore, we use reversible residual layers instead of the standard residuals, which allows storing activations only once in the training process instead of $N$ times, where $N$ is the number of layers. The resulting model, the Reformer, performs on par with Transformer models while being much more memory-efficient and much faster on long sequences.

ICLR 2020

References in corpus (9)

• Generating Long Sequences with Sparse Transformers
• One-shot Learning with Memory-Augmented Neural Networks
• Stand-Alone Self-Attention in Vision Models
• Practical and Optimal LSH for Angular Distance
• Large Memory Layers with Product Keys
• Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
• Hierarchical Memory Networks
• Augmenting Self-attention with Persistent Memory
• Low-Memory Neural Network Training: A Technical Report

Cited by in corpus (172)

• Transformers in Vision: A Survey
• On the Opportunities and Risks of Foundation Models
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Linformer: Self-Attention with Linear Complexity
• Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting
• Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Multi-stage Attention ResU-Net for Semantic Segmentation of Fine-Resolution Remote Sensing Images
• Big Bird: Transformers for Longer Sequences
• SCINet: Time Series Modeling and Forecasting with Sample Convolution and Interaction
• Efficient Training of Audio Transformers with Patchout
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Long Range Arena: A Benchmark for Efficient Transformers
• Residual Attention U-Net for Automated Multi-Class Segmentation of COVID-19 Chest CT Images
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• HiPPO: Recurrent Memory with Optimal Polynomial Projections
• A Practical Survey on Faster and Lighter Transformers
• Perceiver: General Perception with Iterative Attention
• Random Feature Attention
• Rethinking Attention with Performers
• Aligning AI With Shared Human Values
• Beyond 512 Tokens: Siamese Multi-depth Transformer-based Hierarchical Encoder for Long-Form Document Matching
• Sparse DETR: Efficient End-to-End Object Detection with Learnable Sparsity
• Fastformer: Additive Attention Can Be All You Need
• Sparse Sinkhorn Attention
• LoFTR: Detector-Free Local Feature Matching with Transformers
• SOFT: Softmax-free Transformer with Linear Complexity
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• Long-Short Transformer: Efficient Transformers for Language and Vision
• Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
• Luna: Linear Unified Nested Attention
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• DeLighT: Deep and Light-weight Transformer
• An Attention Free Transformer
• The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
• Container: Context Aggregation Network
• Neural ranking models for document retrieval
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Structured Prediction as Translation between Augmented Natural Languages
• LieTransformer: Equivariant self-attention for Lie Groups
• BERT-of-Theseus: Compressing BERT by Progressive Module Replacing
• MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
• Automated Identification of Cell Populations in Flow Cytometry Data with Transformers
• CCVS: Context-aware Controllable Video Synthesis
• Text Guide: Improving the quality of long text classification by a text selection method based on feature importance
• Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention
• Combiner: Full Attention Transformer with Sparse Computation Cost
• A Survey of Deep Learning Approaches for OCR and Document Understanding
• Mind the Gap: Assessing Temporal Generalization in Neural Language Models
• Data Movement Is All You Need: A Case Study on Optimizing Transformers
• Multi-head or Single-head? An Empirical Comparison for Transformer Training
• Neural Language Generation: Formulation, Methods, and Evaluation
• GMAT: Global Memory Augmentation for Transformers
• Recurrent Quantum Neural Networks
• Multi-View Stereo with Transformer
• Findings of the WMT 2024 Shared Task on Chat Translation
• Centroid Transformers: Learning to Abstract with Attention
• RealFormer: Transformer Likes Residual Attention
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
• Multitask Balanced and Recalibrated Network for Medical Code Prediction
• Conformer-Kernel with Query Term Independence for Document Retrieval
• Time-aware Large Kernel Convolutions
• SAC: Accelerating and Structuring Self-Attention via Sparse Adaptive Connection
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• lamBERT: Language and Action Learning Using Multimodal BERT
• MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• Efficient Attentions for Long Document Summarization
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• LiftFormer: 3D Human Pose Estimation using attention models
• Evolving Attention with Residual Convolutions
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• 3D Object Detection with Pointformer
• Generating Images with Sparse Representations
• Noise-Robust End-to-End Quantum Control using Deep Autoregressive Policy Networks
• 3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers
• Co-BERT: A Context-Aware BERT Retrieval Model Incorporating Local and Query-specific Context
• Scatterbrain: Unifying Sparse and Low-rank Attention Approximation
• Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding
• LAMPRET: Layout-Aware Multimodal PreTraining for Document Understanding
• Fast Transformers with Clustered Attention
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• Multiresolution and Multimodal Speech Recognition with Transformers
• Shifted Chunk Transformer for Spatio-Temporal Representational Learning
• SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task
• On the Expressive Power of Self-Attention Matrices
• Weak-Attention Suppression For Transformer Based Speech Recognition
• What Context Features Can Transformer Language Models Use?
• Not All Memories are Created Equal: Learning to Forget by Expiring
• FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
• UFO-BLO: Unbiased First-Order Bilevel Optimization
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• Sub-Linear Memory: How to Make Performers SLiM
• Which *BERT? A Survey Organizing Contextualized Encoders
• Transformers for Limit Order Books
• PnP-DETR: Towards Efficient Visual Analysis with Transformers
• Adding Recurrence to Pretrained Transformers for Improved Efficiency and Context Size
• TensorCoder: Dimension-Wise Attention via Tensor Representation for Natural Language Modeling
• A Sliding-Window Approach to Automatic Creation of Meeting Minutes
• Multiple Sclerosis Severity Classification From Clinical Text
• Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems
• A Unified Efficient Pyramid Transformer for Semantic Segmentation
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• IOT: Instance-wise Layer Reordering for Transformer Structures
• baller2vec++: A Look-Ahead Multi-Entity Transformer For Modeling Coordinated Agents
• Classifying Long Clinical Documents with Pre-trained Transformers
• Multi-scale Transformer Language Models
• Open Question Answering over Tables and Text
• Staircase Attention for Recurrent Processing of Sequences
• Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling
• General-Purpose User Embeddings based on Mobile App Usage
• N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equations
• EL-Attention: Memory Efficient Lossless Attention for Generation
• Sparse Attention with Linear Units
• On Learning the Transformer Kernel
• Independent Encoder for Deep Hierarchical Unsupervised Image-to-Image Translation
• SMYRF: Efficient Attention using Asymmetric Clustering
• Unlocking Pixels for Reinforcement Learning via Implicit Attention
• Sparse Factorization of Large Square Matrices
• Smart Bird: Learnable Sparse Attention for Efficient and Effective Transformer
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• Self-supervised Answer Retrieval on Clinical Notes
• Armour: Generalizable Compact Self-Attention for Vision Transformers
• Dynamic Tensor Rematerialization
• PSEUDo: Interactive Pattern Search in Multivariate Time Series with Locality-Sensitive Hashing and Relevance Feedback
• Adaptive Semiparametric Language Models
• H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
• Token Pooling in Vision Transformers
• Learning Elastic Embeddings for Customizing On-Device Recommenders
• PairConnect: A Compute-Efficient MLP Alternative to Attention
• When Attention Meets Fast Recurrence: Training Language Models with Reduced Compute
• $\infty$-former: Infinite Memory Transformer
• Token Shift Transformer for Video Classification
• Metric Transforms and Low Rank Matrices via Representation Theory of the Real Hyperrectangle
• Do Long-Range Language Models Actually Use Long-Range Context?
• Beyond Nyströmformer -- Approximation of self-attention by Spectral Shifting
• Feedback Attention for Cell Image Segmentation
• MATE: Multi-view Attention for Table Transformer Efficiency
• Speech Summarization using Restricted Self-Attention
• Knowledge Transfer by Discriminative Pre-training for Academic Performance Prediction
• A Strong Baseline for Query Efficient Attacks in a Black Box Setting
• Efficient Transformer for Direct Speech Translation
• Vector-Vector-Matrix Architecture: A Novel Hardware-Aware Framework for Low-Latency Inference in NLP Applications
• Contextualize Knowledge Bases with Transformer for End-to-end Task-Oriented Dialogue Systems
• SHAPE: Shifted Absolute Position Embedding for Transformers
• Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Test
• Plot-guided Adversarial Example Construction for Evaluating Open-domain Story Generation
• Learning Hard Retrieval Decoder Attention for Transformers
• Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
• No Answer is Better Than Wrong Answer: A Reflection Model for Document Level Machine Reading Comprehension
• Understanding and Overcoming the Challenges of Efficient Transformer Quantization
• Leveraging redundancy in attention with Reuse Transformers
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization
• Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method
• A light transformer for speech-to-intent applications
• Reciprocal Supervised Learning Improves Neural Machine Translation
• Does Dialog Length matter for Next Response Selection task? An Empirical Study
• Graph Conditioned Sparse-Attention for Improved Source Code Understanding
• Global Interaction Modelling in Vision Transformer via Super Tokens
• A Survey on Green Deep Learning
• Incremental Neural Coreference Resolution in Constant Memory
• Relation/Entity-Centric Reading Comprehension
• Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks
• Ripple Attention for Visual Perception with Sub-quadratic Complexity
• Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers
• Fast offline Transformer-based end-to-end automatic speech recognition for real-world applications
• PermuteFormer: Efficient Relative Position Encoding for Long Sequences