CFP: Efficient Optimization of Intra-Operator Parallelism Plans for Large Model Training

Weifang Hu, Xuanhua Shi, Yunkai Zhang +7

Optimizing the parallel training of large models requires exploring intra-operator parallelism plans for a computation graph that typically contains tens of thousands of primitive…