LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

Xiaonan Nie, Qibin Liu, Fangcheng Fu +6

Larger transformer models always perform better on various tasks but require more costs to scale up the model size. To efficiently enlarge models, the mixture-of-experts (MoE) arch…