Residual Matrix Transformers: Scaling the Size of the Residual Stream

Brian Mak, Jeffrey Flanigan

The residual stream acts as a memory bus where transformer layers both store and access features (Elhage et al., 2021). We consider changing the mechanism for retrieving and storin…