One QK Channel, Many Sources: Tracing Low-Precision Attention Collapse

Shuxiao Xie, Shuyang Xie, Yuan Cao +3

A bfloat16 transformer can train normally, then collapse abruptly. Prior work links collapse to structured attention errors and shows QK normalization disrupts their compounding. D…