Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise

Abstract

We propose an improved convergence analysis technique that characterizes the distributed learning paradigm of federated learning (FL) with imperfect/noisy uplink and downlink communications. Such imperfect communication scenarios arise in the practical deployment of FL in emerging communication systems and protocols. The analysis developed in this paper demonstrates, for the first time, that there is an asymmetry in the detrimental effects of uplink and downlink communications in FL. In particular, the adverse effect of the downlink noise is more severe on the convergence of FL algorithms. Using this insight, we propose improved Signal-to-Noise (SNR) control strategies that, discarding the negligible higher-order terms, lead to a similar convergence rate for FL as in the case of a perfect, noise-free communication channel while incurring significantly less power resources compared to existing solutions. In particular, we establish that to maintain the $O(\frac{1}{\sqrt{K}})$ rate of convergence like in the case of noise-free FL, we need to scale down the uplink and downlink noise by $\Omega (\sqrt{k})$ and $\Omega (k)$ respectively, where $k$ denotes the communication round, $k=1,\dots ,K$. Our theoretical result is further characterized by two major benefits: firstly, it does not assume the somewhat unrealistic assumption of bounded client dissimilarity, and secondly, it only requires smooth non-convex loss functions, a function class better suited for modern machine learning and deep learning models. We also perform extensive empirical analysis to verify the validity of our theoretical findings.

References in corpus (18)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• On the Convergence of FedAvg on Non-IID Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
• FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization
• Over-the-Air Federated Learning from Heterogeneous Data
• Communication Compression for Decentralized Training
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• SGD and Hogwild! Convergence Without the Bounded Gradients Assumption
• ScaleCom: Scalable Sparsified Gradient Compression for Communication-Efficient Distributed Training
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Fast Convergence Algorithm for Analog Federated Learning
• Federated Learning over Noisy Channels: Convergence Analysis and Design Examples
• Communication-Efficient Variance-Reduced Decentralized Stochastic Optimization over Time-Varying Directed Graphs