Sparse Incremental Aggregation in Multi-Hop Federated Learning

Abstract

This paper investigates federated learning (FL) in a multi-hop communication setup, such as in constellations with inter-satellite links. In this setup, part of the FL clients are responsible for forwarding other client's results to the parameter server. Instead of using conventional routing, the communication efficiency can be improved significantly by using in-network model aggregation at each intermediate hop, known as incremental aggregation (IA). Prior works [1] have indicated diminishing gains for IA under gradient sparsification. Here we study this issue and propose several novel correlated sparsification methods for IA. Numerical results show that, for some of these algorithms, the full potential of IA is still available under sparsification without impairing convergence. We demonstrate a 15x improvement in communication efficiency over conventional routing and a 11x improvement over state-of-the-art (SoA) sparse IA.

arXiv commentThis paper is accepted for the 25th IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC) conference

References in corpus (5)

• Sparse Communication for Distributed Gradient Descent
• Ground-Assisted Federated Learning in LEO Satellite Constellations
• Federated Learning in Satellite Constellations
• On-board Federated Learning for Satellite Clusters with Inter-Satellite Links
• Rethinking gradient sparsification as total error minimization