HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusters

Yuejie Wang, Tao Chang, Yuanyuan Zhao +10

Training Large Language Models (LLMs) on heterogeneous clusters presents significant challenges for collective communication, as hardware from multiple vendors introduces diverse n…