58 citations · 114 across the 15 of their papers we have counts for
Showing 2021Show all
2 papers · 1 filter
cs.LG2021
Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences
Yifan Chen, Qi Zeng, Dilek Hakkani-Tur +3
Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. To address this limitation, Li…
cs.LG2021
Skyformer: Remodel Self-Attention with Gaussian Kernel and Nyström Method
Yifan Chen, Qi Zeng, Heng Ji +1
Transformers are expensive to train due to the quadratic time and space complexity in the self-attention mechanism. On the other hand, although kernel machines suffer from the same…