Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers

Amit Ben-Artzy, Roy Schwartz

In decoder-based LLMs, the representation of a given layer serves two purposes: as input to the next layer during the computation of the current token; and as input to the attentio…