Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel

Chuanyang Zheng, Jiankai Sun, Yihang Gao +13

Mixture-of-Experts (MoE) has become a cornerstone in recent state-of-the-art large language models (LLMs). Traditionally, MoE relies on $\mathrm{Softmax}$ as the router score funct…