MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

Zhitian Xie, Yinger Zhang, Chenyi Zhuang +4

The application of mixture-of-experts (MoE) is gaining popularity due to its ability to improve model's performance. In an MoE structure, the gate layer plays a significant role in…