CO2: Efficient Distributed Training with Full Communication-Computation Overlap

Weigao Sun, Zhen Qin, Weixuan Sun +5

The fundamental success of large language models hinges upon the efficacious implementation of large-scale distributed training techniques. Nevertheless, building a vast, high-perf…