On the Necessity of Attention-FFN Split in Vision Transformers

Junhyeok Kim, Jinyeong Kim, Jae Wan Park +1

The standard Transformer architecture relies on a rigid pattern that alternates Attention and Feed-Forward Network (FFN) layers. Despite its widespread adoption, the inductive bias…