BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning

Abstract

Communication between workers and the master node to collect local stochastic gradients is a key bottleneck in a large-scale federated learning system. Various recent works have proposed to compress the local stochastic gradients to mitigate the communication overhead. However, robustness to malicious attacks is rarely considered in such a setting. In this work, we investigate the problem of Byzantine-robust compressed federated learning, where the attacks from Byzantine workers can be arbitrarily malicious. We theoretically point out that different to the attacks-free compressed stochastic gradient descent (SGD), its vanilla combination with geometric median-based robust aggregation seriously suffers from the compression noise in the presence of Byzantine attacks. In light of this observation, we propose to reduce the compression noise with gradient difference compression so as to improve the Byzantine-robustness. We also observe the impact of the intrinsic stochastic noise caused by selecting random samples, and adopt the stochastic average gradient algorithm (SAGA) to gradually eliminate the inner variations of regular workers. We theoretically prove that the proposed algorithm reaches a neighborhood of the optimal solution at a linear convergence rate, and the asymptotic learning error is in the same order as that of the state-of-the-art uncompressed method. Finally, numerical experiments demonstrate the effectiveness of the proposed method.

References in corpus (21)

• Federated Learning: Strategies for Improving Communication Efficiency
• On the Convergence of FedAvg on Non-IID Data
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
• Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• D${}^2$: Decentralized Training over Decentralized Data
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• Distributed Learning with Compressed Gradient Differences
• ByRDiE: Byzantine-resilient distributed coordinate descent for decentralized learning
• Adversary-resilient Distributed and Decentralized Statistical Inference and Machine Learning: An Overview of Recent Advances Under the Byzantine Threat Model
• The Internet of Things: Secure Distributed Inference
• Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients
• A Double Residual Compression Algorithm for Efficient Distributed Learning
• Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets

Cited by in corpus (2)

• Stochastic-Sign SGD for Federated Learning with Theoretical Guarantees
• Resilience in Online Federated Learning: Mitigating Model-Poisoning Attacks via Partial Sharing