Learning Less Is More: Premature Upper-Layer Attention Specialization Hurts Language Model Pretraining

Jinchang Zhu, Jindong Li, Yuwen Hao +3

A causal-decoder block is hierarchical: lower layers build the residual basis that upper layers attend over. We identify a failure mode in GPT pretraining: upper layers commit to s…