On Learning the Transformer Kernel
arXiv:2110.08323
Abstract
In this work we introduce KERNELIZED TRANSFORMER, a generic, scalable, data driven framework for learning the kernel function in Transformers. Our framework approximates the Transformer kernel as a dot product between spectral feature maps and learns the kernel by learning the spectral distribution. This not only helps in learning a generic kernel end-to-end, but also reduces the time and space complexity of Transformers from quadratic to linear. We show that KERNELIZED TRANSFORMERS achieve performance comparable to existing efficient Transformer architectures, both in terms of accuracy as well as computational efficiency. Our study also demonstrates that the choice of the kernel has a substantial impact on performance, and kernel learning variants are competitive alternatives to fixed kernel Transformers, both in long as well as short sequence tasks.
arXiv commentAccepted to TMLR
References in corpus (24)
- Auto-Encoding Variational Bayes
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Longformer: The Long-Document Transformer
- Generating Long Sequences with Sparse Transformers
- Pointer Sentinel Mixture Models
- Reformer: The Efficient Transformer
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- Gaussian Process Kernels for Pattern Discovery and Extrapolation
- Big Bird: Transformers for Longer Sequences
- Synthesizer: Rethinking Self-Attention in Transformer Models
- A Divide-and-Conquer Solver for Kernel Support Vector Machines
- Orthogonal Random Features
- Sparse Sinkhorn Attention
- On GANs and GMMs
- On the Turing Completeness of Modern Neural Network Architectures
- The Unreasonable Effectiveness of Structured Random Orthogonal Embeddings
- Bayesian Nonparametric Kernel-Learning
- On the Error of Random Fourier Features
- ProGen: Language Modeling for Protein Generation
- Recycling Randomness with Structure for Sublinear time Kernel Expansions
- But How Does It Work in Theory? Linear SVM with Random Features
- Implicit Kernel Learning
- ETC: Encoding Long and Structured Inputs in Transformers
- Harmonizable mixture kernels with variational Fourier features