Collective Communication for Distributed LLM Systems: Planning, Runtime Adaptation, and Computation Coordination

Xuebin Song, Menghao Zhang, Yuezheng Liu +5

Distributed large language model (LLM) systems increasingly rely on collective communication primitives such as AllReduce (AR), ReduceScatter (RS), AllGather (AG), and AlltoAll (A2…