MegaBlocks: Efficient Sparse Training with Mixture-of-Experts

Trevor Gale, Deepak Narayanan, Cliff Young +1

We present MegaBlocks, a system for efficient Mixture-of-Experts (MoE) training on GPUs. Our system is motivated by the limitations of current frameworks, which restrict the dynami…