Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers

Abstract

Transformer models have achieved state-of-the-art results across a diverse range of domains. However, concern over the cost of training the attention mechanism to learn complex dependencies between distant inputs continues to grow. In response, solutions that exploit the structure and sparsity of the learned attention matrix have blossomed. However, real-world applications that involve long sequences, such as biological sequence analysis, may fall short of meeting these assumptions, precluding exploration of these models. To address this challenge, we present a new Transformer architecture, Performer, based on Fast Attention Via Orthogonal Random features (FAVOR). Our mechanism scales linearly rather than quadratically in the number of tokens in the sequence, is characterized by sub-quadratic space complexity and does not incorporate any sparsity pattern priors. Furthermore, it provides strong theoretical guarantees: unbiased estimation of the attention matrix and uniform convergence. It is also backwards-compatible with pre-trained regular Transformers. We demonstrate its effectiveness on the challenging task of protein sequence modeling and provide detailed theoretical analysis.

arXiv commentThis arXiv submission has been deprecated. Please see "Rethinking Attention with Performers" at arXiv:2009.14794 for the most updated version of the paper

References in corpus (15)

• Longformer: The Long-Document Transformer
• Improved Training of Wasserstein GANs
• Identification of direct residue contacts in protein-protein interaction by message passing
• Generating Long Sequences with Sparse Transformers
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Universal Transformers
• Reformer: The Efficient Transformer
• Inferring interaction partners from protein sequences
• Orthogonal Random Features
• The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
• ProGen: Language Modeling for Protein Generation
• Imputer: Sequence Modelling via Imputation and Dynamic Programming
• Orthogonal Estimation of Wasserstein Distances
• Faster Transformer Decoding: N-gram Masked Self-Attention
• Universal Transforming Geometric Network

Cited by in corpus (14)

• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Long Range Arena: A Benchmark for Efficient Transformers
• Fastformer: Additive Attention Can Be All You Need
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Learnable Fourier Features for Multi-Dimensional Spatial Positional Encoding
• Cluster-Former: Clustering-based Sparse Transformer for Long-Range Dependency Encoding
• Efficient conformer-based speech recognition with linear attention
• Not All Memories are Created Equal: Learning to Forget by Expiring
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• FNetAR: Mixing Tokens with Autoregressive Fourier Transforms
• H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
• Reservoir Transformers
• THG: Transformer with Hyperbolic Geometry
• Fixed-Length Protein Embeddings using Contextual Lenses