CommFuse: Hiding Tail Latency via Communication Decomposition and Fusion for Distributed LLM Training

Rezaul Karim, Austin Wen, Wang Zongzuo +3

The rapid growth in the size of large language models has necessitated the partitioning of computational workloads across accelerators such as GPUs, TPUs, and NPUs. However, these…