Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment

Haoyang Li, Fangcheng Fu, Sheng Lin +8

To optimize large Transformer model training, both efficient parallel computing and advanced data management are indispensable. However, current methods often assume a stable and u…