Echo: Error-Compensated 4-Bit Tensor-Parallel Communication for LLM Training and Inference

Xingyu Xie, Xiaoyong Xie, Zhouchen Lin +1

Tensor parallelism ends every layer with an all-reduce that sits between two dependent matrix multiplications, so its cost cannot be hidden and grows as compute outpaces interconne…