Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets

Abstract

This work investigates fault-resilient federated learning when the data samples are non-uniformly distributed across workers, and the number of faulty workers is unknown to the central server. In the presence of adversarially faulty workers who may strategically corrupt datasets, the local messages exchanged (e.g., local gradients and/or local model parameters) can be unreliable, and thus the vanilla stochastic gradient descent (SGD) algorithm is not guaranteed to converge. Recently developed algorithms improve upon vanilla SGD by providing robustness to faulty workers at the price of slowing down convergence. To remedy this limitation, the present work introduces a fault-resilient proximal gradient (FRPG) algorithm that relies on Nesterov's acceleration technique. To reduce the communication overhead of FRPG, a local (L) FRPG algorithm is also developed to allow for intermittent server-workers parameter exchanges. For strongly convex loss functions, FRPG and LFRPG have provably faster convergence rates than a benchmark robust stochastic aggregation algorithm. Moreover, LFRPG converges faster than FRPG while using the same communication rounds. Numerical tests performed on various real datasets confirm the accelerated convergence of FRPG and LFRPG over the robust stochastic aggregation benchmark and competing alternatives.

submitted for possible publication, and based on our independent work

References in corpus (9)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Robust Federated Learning in a Heterogeneous Environment
• A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
• Zeno: Distributed Stochastic Gradient Descent with Suspicion-based Fault-tolerance
• Proximity Without Consensus in Online Multi-Agent Optimization
• First Analysis of Local GD on Heterogeneous Data
• Communication-Efficient Distributed Learning via Lazily Aggregated Quantized Gradients
• Lipschitz constant estimation of Neural Networks via sparse polynomial optimization

Cited by in corpus (4)

• Learning from History for Byzantine Robust Optimization
• BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
• Byzantine-Robust Variance-Reduced Federated Learning over Distributed Non-i.i.d. Data
• Learning to Transmit with Provable Guarantees in Wireless Federated Learning