192 citations · 585 across the 32 of their papers we have counts for
4 papers · 1 filter
Who Says Elephants Can't Run: Bringing Large Scale MoE Models into Cloud Scale Production
Young Jin Kim, Rawn Henry, Raffy Fahim +1
Mixture of Experts (MoE) models with conditional execution of sparsely activated layers have enabled training models with a much larger number of parameters. As a result, these mod…
Language Tokens: A Frustratingly Simple Approach Improves Zero-Shot Performance of Multilingual Translation
Muhammad ElNokrashy, Amr Hendy, Mohamed Maher +2
This paper proposes a simple yet effective method to improve direct (X-to-Y) translation for both cases: zero-shot and when direct data is available. We modify the input tokens at…
Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization
Pengcheng He, Baolin Peng, Liyang Lu +11
This paper presents Z-Code++, a new pre-trained language model optimized for abstractive text summarization. The model extends the state of the art encoder-decoder model using thre…
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers
Rui Liu, Young Jin Kim, Alexandre Muzio +1
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in mod…