Attention Sorting Combats Recency Bias In Long Context Language Models

Alexander Peysakhovich, Adam Lerer

Current language models often fail to incorporate long contexts efficiently during generation. We show that a major contributor to this issue are attention priors that are likely l…