MiCS: Near-linear Scaling for Training Gigantic Model on Public Cloud

Zhen Zhang, Shuai Zheng, Yida Wang +5

Existing general purpose frameworks for gigantic model training, i.e., dense models with billions of parameters, cannot scale efficiently on cloud environment with various networki…