Over-the-Air Federated Learning from Heterogeneous Data

Abstract

Federated learning (FL) is a framework for distributed learning of centralized models. In FL, a set of edge devices train a model using their local data, while repeatedly exchanging their trained updates with a central server. This procedure allows tuning a centralized model in a distributed fashion without having the users share their possibly private data. In this paper, we focus on over-the-air (OTA) FL, which has been suggested recently to reduce the communication overhead of FL due to the repeated transmissions of the model updates by a large number of users over the wireless channel. In OTA FL, all users simultaneously transmit their updates as analog signals over a multiple access channel, and the server receives a superposition of the analog transmitted signals. However, this approach results in the channel noise directly affecting the optimization procedure, which may degrade the accuracy of the trained model. We develop a Convergent OTA FL (COTAF) algorithm which enhances the common local stochastic gradient descent (SGD) FL algorithm, introducing precoding at the users and scaling at the server, which gradually mitigates the effect of the noise. We analyze the convergence of COTAF to the loss minimizing model and quantify the effect of a statistically heterogeneous setup, i.e. when the training data of each user obeys a different distribution. Our analysis reveals the ability of COTAF to achieve a convergence rate similar to that achievable over error-free channels. Our simulations demonstrate the improved convergence of COTAF over vanilla OTA local SGD for training using non-synthetic datasets. Furthermore, we numerically show that the precoding induced by COTAF notably improves the convergence rate and the accuracy of models trained via OTA FL.

arXiv comment15 pages, 8 figures. Part of the results were accepted to be presented in the 2020 IEEE Global Communications Conference (GLOBECOM)

References in corpus (19)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning with Non-IID Data
• On the Convergence of FedAvg on Non-IID Data
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Sparse Communication for Distributed Gradient Descent
• UVeQFed: Universal Vector Quantization for Federated Learning
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Sparsified SGD with Memory
• Local SGD Converges Fast and Communicates Little
• On Analog Gradient Descent Learning over Multiple Access Fading Channels
• Federated Learning: A Signal Processing Perspective
• Over-the-air Function Computation in Sensor Networks
• Communication Efficient Federated Learning over Multiple Access Channels
• Checkpoint Ensembles: Ensemble Methods from a Single Training Process
• Federated Learning in Multi-RIS Aided Systems
• Spectrum and Energy Efficient Multiple Access for Detection in Wireless Sensor Networks
• Privacy For Free: Wireless Federated Learning Via Uncoded Transmission With Adaptive Power Control

Cited by in corpus (25)

• Edge Learning for B5G Networks with Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing
• UVeQFed: Universal Vector Quantization for Federated Learning
• Federated Learning: A Signal Processing Perspective
• STAR-RIS Integrated Non-Orthogonal Multiple Access and Over-the-Air Federated Learning: Framework, Analysis, and Optimization
• Joint Privacy Enhancement and Quantization in Federated Learning
• Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning
• Edge Federated Learning Via Unit-Modulus Over-The-Air Computation
• Combating Interference for Over-the-Air Federated Learning: A Statistical Approach via RIS
• Spectrum Breathing: Protecting Over-the-Air Federated Learning Against Interference
• Bayesian Federated Learning over Wireless Networks
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Beamforming and Device Selection Design in Federated Learning with Over-the-air Aggregation
• Fast Convergence Algorithm for Analog Federated Learning
• Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise
• Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraints
• An Element-Wise Weights Aggregation Method for Federated Learning
• Sparse Training for Federated Learning with Regularized Error Correction
• Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers
• Bayesian Over-The-Air Computation
• Federated Multi-Armed Bandits Under Byzantine Attacks
• Federated Edge Learning with Misaligned Over-The-Air Computation
• Mixed-Precision Federated Learning via Multi-Precision Over-The-Air Aggregation
• AirMixML: Over-the-Air Data Mixup for Inherently Privacy-Preserving Edge Machine Learning
• Channel-Adaptive Robust Aggregation for Over-the-Air Federated Learning in Heterogeneous Networks