ID Balancing: Stable Training of Extremely Sparse MoE via PID-Based Load Control

Peng Jin, Zihan Qiu, Zekun Wang +8

Scaling Large Language Models (LLMs) via Mixture-of-Experts (MoE) enables massive parameter growth with nearly constant per-token computation. However, further scaling the paramete…