32 citations · 80 across the 14 of their papers we have counts for
4 papers · 1 filter
Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training
Shenggui Li, Hongxin Liu, Zhengda Bian +5
The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice…
Maximizing Parallelism in Distributed Training for Huge Neural Networks
Zhengda Bian, Qifan Xu, Boxiang Wang +1
The recent Natural Language Processing techniques have been refreshing the state-of-the-art performance at an incredible speed. Training huge language models is therefore an impera…
Partially Interpretable Estimators (PIE): Black-Box-Refined Interpretable Machine Learning
Tong Wang, Jingyi Yang, Yunyi Li +1
We propose Partially Interpretable Estimators (PIE) which attribute a prediction to individual features via an interpretable model, while a (possibly) small part of the PIE predict…
Tesseract: Parallelize the Tensor Parallelism Efficiently
Boxiang Wang, Qifan Xu, Zhengda Bian +1
Together with the improvements in state-of-the-art accuracies of various tasks, deep learning models are getting significantly larger. However, it is extremely difficult to impleme…