88 citations · 93 across the 5 of their papers we have counts for
5 papers
Scalable Training of Mixture-of-Experts Models with Megatron Core
Zijie Yan, Hongxiao Bai, Xin Yao +42
Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total pa…
MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
Dennis Liu, Zijie Yan, Xin Yao +15
Mixture of Experts (MoE) models enhance neural network scalability by dynamically selecting relevant experts per input token, enabling larger model sizes while maintaining manageab…
Cross-lingual Supervision Improves Unsupervised Neural Machine Translation
Mingxuan Wang, Hongxiao Bai, Hai Zhao +1
Neural machine translation~(NMT) is ineffective for zero-resource languages. Recent works exploring the possibility of unsupervised neural machine translation (UNMT) with only mono…
Memorizing All for Implicit Discourse Relation Recognition
Hongxiao Bai, Hai Zhao, Junhan Zhao
Implicit discourse relation recognition is a challenging task due to the absence of the necessary informative clue from explicit connectives. The prediction of relations requires a…
Deep Enhanced Representation for Implicit Discourse Relation Recognition
Hongxiao Bai, Hai Zhao
Implicit discourse relation recognition is a challenging task as the relation prediction without explicit connectives in discourse parsing needs understanding of text spans and can…