UMoE: Unifying Attention and FFN with Shared Experts

Yuanhang Yang, Chaozheng Wang, Jing Li

Sparse Mixture of Experts (MoE) architectures have emerged as a promising approach for scaling Transformer models. While initial works primarily incorporated MoE into feed-forward…