Do Value Vectors in Deep Layers Need Context from the Residual Stream?

Muyu He, Yuchen Liu, Qingya Huang +1

The success of the transformer architecture is in large part due to its use of attention layers. An attention layer follows the standard neural network paradigm: it takes the resid…