Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints

Seng Pei Liew, Kenta Shinzato, Yuyang Dong

Modern Mixture-of-Experts (MoE) language models are designed based on total parameters (memory footprint) and active parameters (inference cost). However, we find these two factors…