54 citations · 147 across the 11 of their papers we have counts for
13 papers
TorchScale: Transformers at Scale
Shuming Ma, Hongyu Wang, Shaohan Huang +8
Large Transformers have achieved state-of-the-art performance across many tasks. Most open-source libraries on scaling Transformers focus on improving training or inference with be…
A Bilingual Parallel Corpus with Discourse Annotations
Yuchen Eleanor Jiang, Tianyu Liu, Shuming Ma +3
Machine translation (MT) has almost achieved human parity at sentence-level translation. In response, the MT community has, in part, shifted its focus to document-level translation…
Foundation Transformers
Hongyu Wang, Shuming Ma, Shaohan Huang +12
A big convergence of model architectures across language, vision, speech, and multimodal is emerging. However, under the same name "Transformers", the above areas use different imp…
CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation
Jian Yang, Shaohan Huang, Shuming Ma +6
Named entity recognition (NER) suffers from the scarcity of annotated training data, especially for low-resource languages without labeled data. Cross-lingual NER has been proposed…
StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma +4
The Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We point out that existing learning-to-route MoE metho…
DeepNet: Scaling Transformers to 1,000 Layers
Hongyu Wang, Shuming Ma, Li Dong +3
In this paper, we propose a simple yet effective method to stabilize extremely deep Transformers. Specifically, we introduce a new normalization function (DeepNorm) to modify the r…