2 papers
cs.DC2025
Synergistic Tensor and Pipeline Parallelism
Mengshi Qi, Jiaxuan Peng, Jie Zhang +3
In the machine learning system, the hybrid model parallelism combining tensor parallelism (TP) and pipeline parallelism (PP) has become the dominant solution for distributed traini…
cs.DC2025
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
Yujie Wang, Shenhan Zhu, Fangcheng Fu +6
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components. However…