Why don't Graph Transformers attend to the same token?

Alexandre Bismuth

In large language models (LLMs), attention concentrates heavily on the first token. This 'attention sink' is believed to protect deep models against over-mixing. Even though graph…