Layer-wise Positional Bias in Short-Context Language Modeling

Maryam Rahimi, Mahdi Nouri, Yadollah Yaghoobzadeh

Transformer language models systematically prefer tokens at specific input positions regardless of semantic relevance---a phenomenon known as positional bias. Prior work characteri…