UVeQFed: Universal Vector Quantization for Federated Learning

Abstract

Traditional deep learning models are trained at a centralized server using labeled data samples collected from end devices or users. Such data samples often include private information, which the users may not be willing to share. Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. A major challenge that arises in this method is the need of each user to efficiently transmit its learned model over the throughput limited uplink channel. In this work, we tackle this challenge using tools from quantization theory. In particular, we identify the unique characteristics associated with conveying trained models over rate-constrained channels, and propose a suitable quantization scheme for such settings, referred to as universal vector quantization for FL (UVeQFed). We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion. We then theoretically analyze the distortion, showing that it vanishes as the number of users grows. We also characterize the convergence of models trained with the traditional federated averaging method combined with UVeQFed to the model which minimizes the loss function. Our numerical results demonstrate the gains of UVeQFed over previously proposed methods in terms of both distortion induced in quantization and accuracy of the resulting aggregated model.

References in corpus (17)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Challenges, Methods, and Future Directions
• Federated Learning: Strategies for Improving Communication Efficiency
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• On the Convergence of FedAvg on Non-IID Data
• Towards Federated Learning at Scale: System Design
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Sparse Communication for Distributed Gradient Descent
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Over-the-Air Federated Learning from Heterogeneous Data
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
• Hardware-Limited Task-Based Quantization
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• Natural Compression for Distributed Deep Learning
• Asymptotic Task-Based Quantization with Application to Massive MIMO

Cited by in corpus (25)

• From Distributed Machine Learning to Federated Learning: A Survey
• Over-the-Air Federated Learning from Heterogeneous Data
• Federated Learning: A Signal Processing Perspective
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Federated Learning with Spiking Neural Networks
• Joint Privacy Enhancement and Quantization in Federated Learning
• Communication Efficiency in Federated Learning: Achievements and Challenges
• Task-Based Analog-to-Digital Converters
• Combating Interference for Over-the-Air Federated Learning: A Statistical Approach via RIS
• AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices
• FedVQCS: Federated Learning via Vector Quantized Compressed Sensing
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data
• Prompt-enhanced Federated Content Representation Learning for Cross-domain Recommendation
• FedGreen: Federated Learning with Fine-Grained Gradient Compression for Green Mobile Edge Computing
• Bayesian Federated Learning over Wireless Networks
• Device Heterogeneity in Federated Learning: A Superquantile Approach
• Federated Learning with Superquantile Aggregation for Heterogeneous Data
• Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compression
• Quantized and Asynchronous Federated Learning
• Task-Based Graph Signal Compression
• Federated Learning with Local Differential Privacy: Trade-offs between Privacy, Utility, and Communication
• Digital versus Analog Transmissions for Federated Learning over Wireless Networks
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications