MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models

Xin Ye, Daning Cheng, Boyang Zhang +1

Training large-scale Mixture-of-Experts (MoE) models typically requires high-memory, high-bandwidth GPUs (e.g., A100), and their high cost has become a major barrier to large-model…