10 citations · 12 across the 5 of their papers we have counts for
5 papers
TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training
Chang Chen, Min Li, Zhihua Wu +2
Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in scaling up deep neural networks to an extreme scale. Despite that numerous efforts have been made to im…
GUIM -- General User and Item Embedding with Mixture of Representation in E-commerce
Chao Yang, Ru He, Fangquan Lin +3
Our goal is to build general representation (embedding) for each user and each product item across Alibaba's businesses, including Taobao and Tmall which are among the world's bigg…
SAS: Self-Augmentation Strategy for Language Model Pre-training
Yifei Xu, Jingqiao Zhang, Ru He +4
The core of self-supervised learning for pre-training language models includes pre-training task design as well as appropriate data augmentation. Most data augmentations in languag…
Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedup
Cheng Yang, Shengnan Wang, Chao Yang +3
Pre-trained language models, such as BERT, have achieved significant accuracy gain in many natural language processing tasks. Despite its effectiveness, the huge number of paramete…
CoRe: An Efficient Coarse-refined Training Framework for BERT
Cheng Yang, Shengnan Wang, Yuechuan Li +4
In recent years, BERT has made significant breakthroughs on many natural language processing tasks and attracted great attentions. Despite its accuracy gains, the BERT model genera…