Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air

Abstract

We study federated machine learning (ML) at the wireless edge, where power- and bandwidth-limited wireless devices with local datasets carry out distributed stochastic gradient descent (DSGD) with the help of a remote parameter server (PS). Standard approaches assume separate computation and communication, where local gradient estimates are compressed and transmitted to the PS over orthogonal links. Following this digital approach, we introduce D-DSGD, in which the wireless devices employ gradient quantization and error accumulation, and transmit their gradient estimates to the PS over a multiple access channel (MAC). We then introduce a novel analog scheme, called A-DSGD, which exploits the additive nature of the wireless MAC for over-the-air gradient computation, and provide convergence analysis for this approach. In A-DSGD, the devices first sparsify their gradient estimates, and then project them to a lower dimensional space imposed by the available channel bandwidth. These projections are sent directly over the MAC without employing any digital code. Numerical results show that A-DSGD converges faster than D-DSGD thanks to its more efficient use of the limited bandwidth and the natural alignment of the gradient estimates over the channel. The improvement is particularly compelling at low power and low bandwidth regimes. We also illustrate for a classification problem that, A-DSGD is more robust to bias in data distribution across devices, while D-DSGD significantly outperforms other digital schemes in the literature. We also observe that both D-DSGD and A-DSGD perform better by increasing the number of devices (while keeping the total dataset size constant), showing their ability in harnessing the computation power of edge devices.

arXiv commentIEEE Transactions on Signal Processing, Early Access, Mar. 2020

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Message Passing Algorithms for Compressed Sensing
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Speeding Up Distributed Machine Learning Using Codes
• The dynamics of message passing on dense graphs, with applications to compressed sensing
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• Sparse Communication for Distributed Gradient Descent
• Distributed Federated Learning for Ultra-Reliable Low-Latency Vehicular Communications
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• ATOMO: Communication-efficient Learning via Atomic Sparsification
• Capacity-achieving Sparse Superposition Codes via Approximate Message Passing Decoding
• Federated Multi-Task Learning
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Variance-based Gradient Compression for Efficient Distributed Deep Learning
• Federated Learning via Over-the-Air Computation
• Robust Gradient Descent via Moment Encoding with LDPC Codes
• SparseCast: Hybrid Digital-Analog Wireless Image Transmission Exploiting Frequency Domain Sparsity
• Slow and Stale Gradients Can Win the Race: Error-Runtime Trade-offs in Distributed SGD
• The Error Probability of Sparse Superposition Codes with Approximate Message Passing Decoding
• The Convergence of Sparsified Gradient Methods
• Near-Optimal Straggler Mitigation for Distributed Gradient Methods

Cited by in corpus (99)

• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks
• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• Over-the-Air Federated Learning from Heterogeneous Data
• Federated Learning via Intelligent Reflecting Surface
• On Analog Gradient Descent Learning over Multiple Access Fading Channels
• Federated Learning: A Signal Processing Perspective
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Interference Management for Over-the-Air Federated Learning in Multi-Cell Wireless Networks
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing
• STAR-RIS Integrated Non-Orthogonal Multiple Access and Over-the-Air Federated Learning: Framework, Analysis, and Optimization
• Over-the-Air Computation for 6G: Foundations, Technologies, and Applications
• Integrating Over-the-Air Federated Learning and Non-Orthogonal Multiple Access: What Role can RIS Play?
• Joint Optimization of Communications and Federated Learning Over the Air
• Edge Federated Learning Via Unit-Modulus Over-The-Air Computation
• Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning
• Communication Efficient Federated Learning over Multiple Access Channels
• Balancing Accuracy and Integrity for Reconfigurable Intelligent Surface-aided Over-the-Air Federated Learning
• Near-Field Terahertz Communications: Model-Based and Model-Free Channel Estimation
• Energy-Efficient Radio Resource Allocation for Federated Edge Learning
• Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• Combating Interference for Over-the-Air Federated Learning: A Statistical Approach via RIS
• Convergence Time Optimization for Federated Learning over Wireless Networks
• Federated Learning With Quantized Global Model Updates
• FedVQCS: Federated Learning via Vector Quantized Compressed Sensing
• BEV-SGD: Best Effort Voting SGD for Analog Aggregation Based Federated Learning against Byzantine Attackers
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Over-the-Air Computing for Wireless Data Aggregation in Massive IoT
• Wireless Federated Learning with Local Differential Privacy
• Over-The-Air Computation in Correlated Channels
• Spectrum Breathing: Protecting Over-the-Air Federated Learning Against Interference
• Over-The-Air Computation in Correlated Channels
• The Magic of Superposition: A Survey on Simultaneous Transmission Based Wireless Systems
• Denoising Noisy Neural Networks: A Bayesian Approach with Compensation
• Federated Over-Air Subspace Tracking from Incomplete and Corrupted Data
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Gradient Statistics Aware Power Control for Over-the-Air Federated Learning
• Learning Rate Optimization for Federated Learning Exploiting Over-the-air Computation
• Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
• Joint Device Scheduling and Resource Allocation for Latency Constrained Wireless Federated Learning
• Optimized Power Control Design for Over-the-Air Federated Edge Learning
• Device Scheduling with Fast Convergence for Wireless Federated Learning
• Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challenges
• One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis
• Beamforming and Device Selection Design in Federated Learning with Over-the-air Aggregation
• Private Collaborative Edge Inference via Over-the-Air Computation
• Wireless for Machine Learning
• Privacy For Free: Wireless Federated Learning Via Uncoded Transmission With Adaptive Power Control
• Temporal-Structure-Assisted Gradient Aggregation for Over-the-Air Federated Edge Learning
• Wireless Data Acquisition for Edge Learning: Data-Importance Aware Retransmission
• Reconfigurable Intelligent Surface Enabled Federated Learning: A Unified Communication-Learning Design Approach
• Over-the-Air Computation Systems: Optimization, Analysis and Scaling Laws
• Fast Convergence Algorithm for Analog Federated Learning
• Cost-Effective Federated Learning in Mobile Edge Networks
• Energy-Efficient Resource Management for Federated Edge Learning with CPU-GPU Heterogeneous Computing
• Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraints
• Data-Importance Aware User Scheduling for Communication-Efficient Edge Machine Learning
• On the Communication Latency of Wireless Decentralized Learning
• Wirelessly Powered Federated Edge Learning: Optimal Tradeoffs Between Convergence and Power Transfer
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• An Energy and Carbon Footprint Analysis of Distributed and Federated Learning
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• Over-the-Air Federated Learning with Retransmissions (Extended Version)
• Sparse Training for Federated Learning with Regularized Error Correction
• 1-Bit Compressive Sensing for Efficient Federated Learning Over the Air
• A Compressive Sensing Approach for Federated Learning over Massive MIMO Communication Systems
• Over-the-Air Computation via Reconfigurable Intelligent Surface
• Deploying Federated Learning in Large-Scale Cellular Networks: Spatial Convergence Analysis
• Over-the-Air Computation Systems: Optimal Design with Sum-Power Constraint
• Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers
• Cooperative Learning via Federated Distillation over Fading Channels
• Simultaneous Computation and Communication over MAC
• Bayesian Over-The-Air Computation
• Federated Edge Learning with Misaligned Over-The-Air Computation
• Federated Learning over Wireless Networks: A Band-limited Coordinated Descent Approach
• Start Making Sense: Semantic Plane Filtering and Control for Post-5G Connectivity
• Wireless Networked Control Systems with Coding-Free Data Transmission for Industrial IoT
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels
• Nothing Wasted: Full Contribution Enforcement in Federated Edge Learning
• Unit-Modulus Wireless Federated Learning Via Penalty Alternating Minimization
• Distributed U-net model and Image Segmentation for Lung Cancer Detection
• Over-the-Air Computation via Broadband Channels
• Revolutionizing Wireless Networks with Federated Learning: A Comprehensive Review
• Adaptive Federated Learning With Gradient Compression in Uplink NOMA
• Distributed Sparse SGD with Majority Voting
• Optimising cost vs accuracy of decentralised analytics in fog computing environments
• Spatio-Temporal Federated Learning for Massive Wireless Edge Networks
• Integrated user scheduling and beam steering in over-the-air federated learning for mobile IoT
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• Federated Channel Learning for Intelligent Reflecting Surfaces With Fewer Pilot Signals
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Strong Privacy and Utility Guarantee: Over-the-Air Statistical Estimation
• Decentralized Wireless Federated Learning with Differential Privacy
• Over-the-Air Computation with Spatial-and-Temporal Correlated Signals
• Wireless Distributed Edge Learning: How Many Edge Devices Do We Need?
• Over-the-Air Decentralized Federated Learning
• Byzantine-Resilient Federated Machine Learning via Over-the-Air Computation