HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Models

Si Xu, Zixiao Huang, Yan Zeng +11

Training large-scale models relies on a vast number of computing resources. For example, training the GPT-4 model (1.8 trillion parameters) requires 25000 A100 GPUs . It is a chall…