On Separate Normalization in Self-supervised Transformers

Xiaohui Chen, Yinkai Wang, Yuanqi Du +2

Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (M…