ResiDual Transformer Alignment with Spectral Decomposition

Lorenzo Basile, Valentino Maiorca, Luca Bortolussi +2

When examined through the lens of their residual streams, a puzzling property emerges in transformer networks: residual contributions (e.g., attention heads) sometimes specialize i…