SATFormer: Learning When to Use Early Value Residuals in Transformers

Skye Gunasekaran, Téa Wright, Rui-Jie Zhu +1

Recent Transformer architectures expose later layers to representations computed in earlier layers, motivated by the observation that low-level features can become harder to recove…