High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning

Abstract

Edge machine learning involves the deployment of learning algorithms at the wireless network edge so as to leverage massive mobile data for enabling intelligent applications. The mainstream edge learning approach, federated learning, has been developed based on distributed gradient descent. Based on the approach, stochastic gradients are computed at edge devices and then transmitted to an edge server for updating a global AI model. Since each stochastic gradient is typically high-dimensional (with millions to billions of coefficients), communication overhead becomes a bottleneck for edge learning. To address this issue, we propose in this work a novel framework of hierarchical stochastic gradient quantization and study its effect on the learning performance. First, the framework features a practical hierarchical architecture for decomposing the stochastic gradient into its norm and normalized block gradients, and efficiently quantizes them using a uniform quantizer and a low-dimensional codebook on a Grassmann manifold, respectively. Subsequently, the quantized normalized block gradients are scaled and cascaded to yield the quantized normalized stochastic gradient using a so-called hinge vector designed under the criterion of minimum distortion. The hinge vector is also efficiently compressed using another low-dimensional Grassmannian quantizer. The other feature of the framework is a bit-allocation scheme for reducing the quantization error. The scheme determines the resolutions of the low-dimensional quantizers in the proposed framework. The framework is proved to guarantee model convergency by analyzing the convergence rate as a function of the quantization bits. Furthermore, by simulation, our design is shown to substantially reduce the communication overhead compared with the state-of-the-art signSGD scheme, while both achieve similar learning accuracies.

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Federated Learning: Strategies for Improving Communication Efficiency
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Noncoherent Trellis Coded Quantization: A Practical Limited Feedback Technique for Massive MIMO Systems
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• Towards an Intelligent Edge: Wireless Communication Meets Machine Learning
• Directional Analysis of Stochastic Gradient Descent via von Mises-Fisher Distributions in Deep learning

Cited by in corpus (25)

• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• Hierarchical Federated Learning with Quantization: Convergence Analysis and System Design
• Edge Federated Learning Via Unit-Modulus Over-The-Air Computation
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• FedVQCS: Federated Learning via Vector Quantized Compressed Sensing
• Communication-Efficient Edge AI: Algorithms and Systems
• Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregation
• Privacy For Free: Wireless Federated Learning Via Uncoded Transmission With Adaptive Power Control
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise
• Dynamic Scheduling for Over-the-Air Federated Edge Learning with Energy Constraints
• Wirelessly Powered Federated Edge Learning: Optimal Tradeoffs Between Convergence and Power Transfer
• A Perspective on Time towards Wireless 6G
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• Reconfigurable Intelligent Surface Assisted Mobile Edge Computing with Heterogeneous Learning Tasks
• TAGC: Optimizing Gradient Communication in Distributed Transformer Training
• An Introduction to Communication Efficient Edge Machine Learning
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• Cooperative Learning via Federated Distillation over Fading Channels
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels
• Unit-Modulus Wireless Federated Learning Via Penalty Alternating Minimization
• Edge Learning with Timeliness Constraints: Challenges and Solutions
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Smoothed Differential Privacy
• Design and Analysis of Uplink and Downlink Communications for Federated Learning