Orthogonal Quadratic Complements for Vision Transformer Feed-Forward Networks

Wang Zixian

Recent bilinear feed-forward replacements for vision transformers can substantially improve accuracy, but they often conflate two effects: stronger second-order interactions and in…