FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks

Maksim Zubkov, Daniil Gavrilov

Transformers achieve remarkable performance in various domains, including NLP, CV, audio processing, and graph analysis. However, they do not scale well on long sequence tasks due…