FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization

Abstract

Federated learning is a distributed framework according to which a model is trained over a set of devices, while keeping data localized. This framework faces several systems-oriented challenges which include (i) communication bottleneck since a large number of devices upload their local updates to a parameter server, and (ii) scalability as the federated network consists of millions of devices. Due to these systems challenges as well as issues related to statistical heterogeneity of data and privacy concerns, designing a provably efficient federated learning method is of significant importance yet it remains challenging. In this paper, we present FedPAQ, a communication-efficient Federated Learning method with Periodic Averaging and Quantization. FedPAQ relies on three key features: (1) periodic averaging where models are updated locally at devices and only periodically averaged at the server; (2) partial device participation where only a fraction of devices participate in each round of the training; and (3) quantized message-passing where the edge nodes quantize their updates before uploading to the parameter server. These features address the communications and scalability challenges in federated learning. We also show that FedPAQ achieves near-optimal theoretical guarantees for strongly convex and non-convex loss functions and empirically demonstrate the communication-computation tradeoff provided by our method.

References in corpus (21)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• On the Convergence of FedAvg on Non-IID Data
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Speeding Up Distributed Machine Learning Using Codes
• Learning Differentially Private Recurrent Language Models
• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• On First-Order Meta-Learning Algorithms
• Distributed Federated Learning for Ultra-Reliable Low-Latency Vehicular Communications
• Polynomial Codes: an Optimal Design for High-Dimensional Coded Matrix Multiplication
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Robust Federated Learning in a Heterogeneous Environment
• An Exact Quantized Decentralized Gradient Descent Algorithm
• One-Shot Federated Learning
• Peer-to-peer Federated Learning on Graphs
• CoCoA: A General Framework for Communication-Efficient Distributed Optimization
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• Federated Heavy Hitters Discovery with Differential Privacy

Cited by in corpus (48)

• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• FedALA: Adaptive Local Aggregation for Personalized Federated Learning
• Personalized Federated Learning: A Meta-Learning Approach
• UVeQFed: Universal Vector Quantization for Federated Learning
• Personalized Federated Learning with Moreau Envelopes
• Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• Distilled One-Shot Federated Learning
• A Systematic Survey of Blockchained Federated Learning
• Robust Federated Learning: The Case of Affine Distribution Shifts
• Semi-Synchronous Federated Learning for Energy-Efficient Training and Accelerated Convergence in Cross-Silo Settings
• Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• A Federated Deep Learning Framework for Privacy Preservation and Communication Efficiency
• Federated Learning With Quantized Global Model Updates
• AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• A Privacy Preserving System for Movie Recommendations Using Federated Learning
• Federated Accelerated Stochastic Gradient Descent
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• DACFL: Dynamic Average Consensus Based Federated Learning in Decentralized Topology
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• Accelerating Vertical Federated Learning
• Federated Learning for Energy Constrained IoT devices: A systematic mapping study
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Privacy For Free: Wireless Federated Learning Via Uncoded Transmission With Adaptive Power Control
• Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise
• Distributed Fixed Point Methods with Compressed Iterates
• Communication-Efficient Distributed SVD via Local Power Iterations
• On the Benefits of Multiple Gossip Steps in Communication-Constrained Decentralized Optimization
• FedOCR: Communication-Efficient Federated Learning for Scene Text Recognition
• Federated Learning via Plurality Vote
• Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
• On the Convergence of Quantized Parallel Restarted SGD for Central Server Free Distributed Training
• Neural Tangent Kernel Empowered Federated Learning
• Local Methods with Adaptivity via Scaling
• Optimization-Based GenQSGD for Federated Edge Learning
• FedDQ: Communication-Efficient Federated Learning with Descending Quantization
• Hessian-aware Quantized Node Embeddings for Recommendation
• Federated Learning from Small Datasets
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions