Autonomy-of-Experts Models

Ang Lv, Ruobing Xie, Yining Qian +5

Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue t…