Cobalt: Leveraging Expert Co-activation for Efficient Distributed MoE Training

Junkang Zhou, Xinyi Liu, Fangcheng Fu

Mixture-of-Experts (MoE) has increasingly become a mainstream approach for scaling large language models, as it expands model capacity while keeping computation cost nearly constan…