Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion

Binchi Zhang, Zaiyi Zheng, Zhengzhang Chen +1

Symmetry in the parameter space of deep neural networks (DNNs) has proven beneficial for various deep learning applications. A well-known example is the permutation symmetry in Mul…