Federated Learning With Quantized Global Model Updates

Abstract

We study federated learning (FL), which enables mobile devices to utilize their local datasets to collaboratively train a global model with the help of a central server, while keeping data localized. At each iteration, the server broadcasts the current global model to the devices for local training, and aggregates the local model updates from the devices to update the global model. Previous work on the communication efficiency of FL has mainly focused on the aggregation of model updates from the devices, assuming perfect broadcasting of the global model. In this paper, we instead consider broadcasting a compressed version of the global model. This is to further reduce the communication cost of FL, which can be particularly limited when the global model is to be transmitted over a wireless medium. We introduce a lossy FL (LFL) algorithm, in which both the global model and the local model updates are quantized before being transmitted. We analyze the convergence behavior of the proposed LFL algorithm assuming the availability of accurate local model updates at the server. Numerical experiments show that the proposed LFL scheme, which quantizes the global model update (with respect to the global model estimate at the devices) rather than the global model itself, significantly outperforms other existing schemes studying quantization of the global model at the PS-to-device direction. Also, the performance loss of the proposed scheme is marginal compared to the fully lossless approach, where the PS and the devices transmit their messages entirely without any quantization.

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Don't Use Large Mini-Batches, Use Local SGD
• On the Convergence of Local Descent Methods in Federated Learning
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• The Non-IID Data Quagmire of Decentralized Machine Learning
• First Analysis of Local GD on Heterogeneous Data
• COLA: Decentralized Linear Learning
• Faster On-Device Training Using New Federated Momentum Algorithm
• Semi-Cyclic Stochastic Gradient Descent
• The Convergence of Sparsified Gradient Methods
• FedDANE: A Federated Newton-Type Method

Cited by in corpus (10)

• FedML: A Research Library and Benchmark for Federated Machine Learning
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Communication Efficiency in Federated Learning: Achievements and Challenges
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• Communication Efficient Distributed Learning with Censored, Quantized, and Generalized Group ADMM
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• Efficient Ring-topology Decentralized Federated Learning with Deep Generative Models for Industrial Artificial Intelligent
• Server Averaging for Federated Learning
• An Expectation-Maximization Perspective on Federated Learning
• Design and Analysis of Uplink and Downlink Communications for Federated Learning