2 papers
cs.LG2025
Cutting the Skip: Training Residual-Free Transformers
Yiping Ji, James Martens, Jianqiao Zheng +5
Transformers have achieved remarkable success across a wide range of applications, a feat often attributed to their scalability. Yet training them without skip (residual) connectio…
cs.LG2025
Always Skip Attention
Yiping Ji, Hemanth Saratchandran, Peyman Moghadam +1
We highlight a curious empirical result within modern Vision Transformers (ViTs). Specifically, self-attention catastrophically fails to train unless it is used in conjunction with…