Memory-Efficient Expert Routing for Distributed MoE Training

Arnab Kanti Tarafder, Jaume Guasch-Martí, Gokcen Kestor +1

As Mixture-of-Experts (MoE) models scale toward hundreds of experts and higher top-$k$ routing, memory efficiency in distributed training becomes a critical bottleneck. Peak memory…