Publications (8)
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task
Jian Yang, Shuming Ma, Haoyang Huang +8
This report describes Microsoft's machine translation systems for the WMT21 shared task on large-scale multilingual machine translation. We participated in all three evaluation tra…
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders
Shuming Ma, Li Dong, Shaohan Huang +6
While pretrained encoders have achieved success in various natural language understanding (NLU) tasks, there is a gap between these pretrained encoders and natural language generat…
Improving Multilingual Translation by Representation and Gradient Regularization
Yilin Yang, Akiko Eriguchi, Alexandre Muzio +3
Multilingual Neural Machine Translation (NMT) enables one model to serve all translation directions, including ones that are unseen during training, i.e. zero-shot translation. Des…
Discovering Representation Sprachbund For Multilingual Pre-Training
Yimin Fan, Yaobo Liang, Alexandre Muzio +4
Multilingual pre-trained models have demonstrated their effectiveness in many multilingual NLP tasks and enabled zero-shot or few-shot transfer from high-resource languages to low…
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders
Shuming Ma, Jian Yang, Haoyang Huang +10
Multilingual machine translation enables a single model to translate between different languages. Most existing multilingual machine translation systems adopt a randomly initialize…
Gating Dropout: Communication-efficient Regularization for Sparsely Activated Transformers
Rui Liu, Young Jin Kim, Alexandre Muzio +1
Sparsely activated transformers, such as Mixture of Experts (MoE), have received great interest due to their outrageous scaling capability which enables dramatical increases in mod…
SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
Alexandre Muzio, Alex Sun, Churan He
The advancement of deep learning has led to the emergence of Mixture-of-Experts (MoEs) models, known for their dynamic allocation of computational resources based on input. Despite…
Scalable and Efficient MoE Training for Multitask Multilingual Models
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio +6
The Mixture of Experts (MoE) models are an emerging class of sparsely activated deep learning models that have sublinear compute costs with respect to their parameters. In contrast…