Yuan 2.0-M32: Mixture of Experts with Attention Router

Shaohua Wu, Jiangang Luo, Xi Chen +12

Yuan 2.0-M32, with a similar base architecture as Yuan-2.0 2B, uses a mixture-of-experts architecture with 32 experts of which 2 experts are active. A new router network, Attention…