GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory

Haoze Wu, Zihan Qiu, Zili Wang +2

Mixture-of-Experts (MoE) has been demonstrated as an efficient method to scale up models. By dynamically and sparsely selecting activated experts, MoE can effectively reduce comput…