Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

Shenggui Li, Hongxin Liu, Zhengda Bian +5

The success of Transformer models has pushed the deep learning model scale to billions of parameters. Due to the limited memory resource of a single GPU, However, the best practice…