LoCo: Low-Bit Communication Adaptor for Large-scale Model Training

Xingyu Xie, Zhijie Lin, Kim-Chuan Toh +1

To efficiently train large-scale models, low-bit gradient communication compresses full-precision gradients on local GPU nodes into low-precision ones for higher gradient synchroni…