Complete-muE: Optimal Hyperparameter Transfer and Scaling for MoE Models

Hongwu Peng, Ohiremen Dibua, Yuanjun Xiong +8

We propose Complete-muE, a framework which targets hyperparameter transfer across dense FFN and any Mixture-of-Experts (MoE) setups in transformer blocks. Existing tools such as $μ…