Fast and Exact Relative-Position Attention via Structure-Aware Kernels

Delyan Boychev, Ivo Petrov, Dimitar I. Dimitrov

Relative-position attention mechanisms offer compelling alternatives to RoPE across various transformer architectures, delivering superior performance in domains such as audio. The…