Why do LLMs attend to the first token?

Federico Barbero, Álvaro Arroyo, Xiangming Gu +4

Large Language Models (LLMs) tend to attend heavily to the first token in the sequence -- creating a so-called attention sink. Many works have studied this phenomenon in detail, pr…