Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms

Jiahao Ying, Mingbao Lin, Qianru Sun +1

Mixture-of-Experts (MoE) architectures have emerged as a promising direction, offering efficiency and scalability by activating only a subset of parameters during inference. Howeve…