Wyner-Ziv Gradient Compression for Federated Learning

Abstract

Due to limited communication resources at the client and a massive number of model parameters, large-scale distributed learning tasks suffer from communication bottleneck. Gradient compression is an effective method to reduce communication load by transmitting compressed gradients. Motivated by the fact that in the scenario of stochastic gradients descent, gradients between adjacent rounds may have a high correlation since they wish to learn the same model, this paper proposes a practical gradient compression scheme for federated learning, which uses historical gradients to compress gradients and is based on Wyner-Ziv coding but without any probabilistic assumption. We also implement our gradient quantization method on the real dataset, and the performance of our method is better than the previous schemes.

References in corpus (8)

• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Sparsified SGD with Memory
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Distributed Mean Estimation with Limited Communication
• Optimal Gradient Compression for Distributed and Federated Learning
• Breaking the Communication-Privacy-Accuracy Trilemma
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees