Accelerating Heterogeneous Tensor Parallelism via Flexible Workload Control

Zhigang Wang, Xu Zhang, Ning Wang +5

Transformer-based models are becoming deeper and larger recently. For better scalability, an underlying training solution in industry is to split billions of parameters (tensors) i…