Why Attention Patterns Exist: A Unifying Temporal Perspective Analysis

Qingyue Yang, Jie Wang, Xing Li +6

Attention patterns play a crucial role in both training and inference of large language models (LLMs). Prior works have identified individual patterns such as retrieval heads, sink…