Token Weighting for Long-Range Language Modeling

Falko Helm, Nico Daheim, Iryna Gurevych

Many applications of large language models (LLMs) require long-context understanding, but models continue to struggle with such tasks. We hypothesize that conventional next-token p…