192 citations · 488 across the 17 of their papers we have counts for
4 papers · 1 filter
Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
Young Jin Kim, Rawn Henry, Raffy Fahim +1
Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters. As a result, these mod…
AutoMoE: Heterogeneous Mixture-of-Experts with Adaptive Computation for Efficient Neural Machine Translation
Ganesh Jawahar, Subhabrata Mukherjee, Xiaodong Liu +6
Mixture-of-Expert (MoE) models have obtained state-of-the-art performance in Neural Machine Translation (NMT) tasks. Existing works in MoE mostly consider a homogeneous design wher…
Fast Vocabulary Projection Method via Clustering for Multilingual Machine Translation on GPU
Hossam Amer, Young Jin Kim, Mohamed Afify +2
Multilingual Neural Machine Translation has been showing great success using transformer models. Deploying these models is challenging because they usually require large vocabulary…
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers
Rui Liu, Young Jin Kim, Alexandre Muzio +1
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in mod…