Context-aware Biases for Length Extrapolation

Ali Veisi, Hamidreza Amirzadeh, Amir Mansourian

Transformers often struggle to generalize to longer sequences than those seen during training, a limitation known as length extrapolation. Most existing Relative Positional Encodin…