Depth-Attention: Cross-Layer Value Mixing for Language Models

Boyi Zeng, Yiqin Hao, Zitong Wang +7

Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectiv…