Byzantine-Robust Variance-Reduced Federated Learning over Distributed Non-i.i.d. Data

Abstract

We consider the federated learning problem where data on workers are not independent and identically distributed (i.i.d.). During the learning process, an unknown number of Byzantine workers may send malicious messages to the central node, leading to remarkable learning error. Most of the Byzantine-robust methods address this issue by using robust aggregation rules to aggregate the received messages, but rely on the assumption that all the regular workers have i.i.d. data, which is not the case in many federated learning applications. In light of the significance of reducing stochastic gradient noise for mitigating the effect of Byzantine attacks, we use a resampling strategy to reduce the impact of both inner variation (that describes the sample heterogeneity on every regular worker) and outer variation (that describes the sample heterogeneity among the regular workers), along with a stochastic average gradient algorithm to gradually eliminate the inner variation. The variance-reduced messages are then aggregated with a robust geometric median operator. We prove that the proposed method reaches a neighborhood of the optimal solution at a linear convergence rate and the learning error is determined by the number of Byzantine workers. Numerical experiments corroborate the theoretical results and show that the proposed method outperforms the state-of-the-arts in the non-i.i.d. setting.

References in corpus (14)

• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• The Hidden Vulnerability of Distributed Learning in Byzantium
• Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks
• Learning to Detect Malicious Clients for Robust Federated Learning
• Robust Federated Learning in a Heterogeneous Environment
• Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation
• Learning from History for Byzantine Robust Optimization
• Distributed Momentum for Byzantine-resilient Learning
• Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets
• Byzantine Resilient Non-Convex SVRG with Distributed Batch Gradient Computations
• Holdout SGD: Byzantine Tolerant Federated Learning