Position Bias Correction is Insufficient for One-Pass Attention Sorting

Qiong Tang, Xiangkun Hu, Xiangyang Liu +2

Long-context language models suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents…