Quantize Once, Train Fast: Allreduce-Compatible Compression with Provable Guarantees

Jihao Xin, Marco Canini, Peter Richtárik +1

Distributed training enables large-scale deep learning, but suffers from high communication overhead, especially as models and datasets grow. Gradient compression, particularly qua…