Fast Federated Learning by Balancing Communication Trade-Offs

Abstract

Federated Learning (FL) has recently received a lot of attention for large-scale privacy-preserving machine learning. However, high communication overheads due to frequent gradient transmissions decelerate FL. To mitigate the communication overheads, two main techniques have been studied: (i) local update of weights characterizing the trade-off between communication and computation and (ii) gradient compression characterizing the trade-off between communication and precision. To the best of our knowledge, studying and balancing those two trade-offs jointly and dynamically while considering their impacts on convergence has remained unresolved even though it promises significantly faster FL. In this paper, we first formulate our problem to minimize learning error with respect to two variables: local update coefficients and sparsity budgets of gradient compression who characterize trade-offs between communication and computation/precision, respectively. We then derive an upper bound of the learning error in a given wall-clock time considering the interdependency between the two variables. Based on this theoretical analysis, we propose an enhanced FL scheme, namely Fast FL (FFL), that jointly and dynamically adjusts the two variables to minimize the learning error. We demonstrate that FFL consistently achieves higher accuracies faster than similar schemes existing in the literature.

arXiv comment14 pages, 24 figures, accepted for publication in IEEE Transactions on Communications

References in corpus (13)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Don't Use Large Mini-Batches, Use Local SGD
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

Cited by in corpus (5)

• FedKD: Communication Efficient Federated Learning via Knowledge Distillation
• Hierarchical Federated Learning with Quantization: Convergence Analysis and System Design
• Toward Ambient Intelligence: Federated Edge Learning with Task-Oriented Sensing, Computation, and Communication Integration
• Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compression
• Quantized and Asynchronous Federated Learning