L-GreCo: Layerwise-Adaptive Gradient Compression for Efficient and Accurate Deep Learning

Mohammadreza Alimohammadi, Ilia Markov, Elias Frantar +1

Data-parallel distributed training of deep neural networks (DNN) has gained very widespread adoption, but can still experience communication bottlenecks. To address this issue, ent…