Nexusformer: Nonlinear Attention Expansion for Stable and Inheritable Transformer Scaling

Weijie Zhao, Mingquan Liu, Bolun Wang +4

Scaling Transformers typically necessitates training larger models from scratch, as standard architectures struggle to expand without discarding learned representations. We identif…