2 papers
cs.CL2025
Context-aware Biases for Length Extrapolation
Ali Veisi, Hamidreza Amirzadeh, Amir Mansourian
Transformers often struggle to generalize to longer sequences than those seen during training, a limitation known as length extrapolation. Most existing Relative Positional Encodin…
cs.CL2025
Context-aware Rotary Position Embedding
Ali Veisi, Delaram Fartoot, Hamidreza Amirzadeh
Positional encoding is a vital component of Transformer architectures, enabling models to incorporate sequence order into self-attention mechanisms. Rotary Positional Embeddings (R…