29 citations · 83 across the 13 of their papers we have counts for
4 papers · 1 filter
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining
Junyang Lin, An Yang, Jinze Bai +9
Recent expeditious developments in deep learning algorithms, distributed training, and even hardware design for large models have enabled training extreme-scale models, say GPT-3 a…
M6-T: Exploring Sparse Expert Models and Beyond
An Yang, Junyang Lin, Rui Men +12
Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost, and thus it has become a trend in model scal…
Graph Structural-topic Neural Network
Qingqing Long, Yilun Jin, Guojie Song +2
Graph Convolutional Networks (GCNs) achieved tremendous success by effectively gathering local features for nodes. However, commonly do GCNs focus more on node features but less on…
DL2: A Deep Learning-driven Scheduler for Deep Learning Clusters
Yanghua Peng, Yixin Bao, Yangrui Chen +3
More and more companies have deployed machine learning (ML) clusters, where deep learning (DL) models are trained for providing various AI-driven services. Efficient resource sched…