Lazarus: Resilient and Elastic Training of Mixture-of-Experts Models

Yongji Wu, Wenjie Qu, Xueshen Liu +10

Sparsely-activated Mixture-of-Experts (MoE) architecture has increasingly been adopted to further scale large language models (LLMs). However, frequent failures still pose signific…