Route Experts by Sequence, not by Token

Tiansheng Wen, Yifei Wang, Aosong Feng +7

Mixture-of-Experts (MoE) architectures scale large language models (LLMs) by activating only a subset of experts per token, but the standard TopK routing assigns the same fixed num…