Statistical Estimation and Inference via Local SGD in Federated Learning

Abstract

Federated Learning (FL) makes a large amount of edge computing devices (e.g., mobile phones) jointly learn a global model without data sharing. In FL, data are generated in a decentralized manner with high heterogeneity. This paper studies how to perform statistical estimation and inference in the federated setting. We analyze the so-called Local SGD, a multi-round estimation procedure that uses intermittent communication to improve communication efficiency. We first establish a {\it functional central limit theorem} that shows the averaged iterates of Local SGD weakly converge to a rescaled Brownian motion. We next provide two iterative inference methods: the {\it plug-in} and the {\it random scaling}. Random scaling constructs an asymptotically pivotal statistic for inference by using the information along the whole Local SGD path. Both the methods are communication efficient and applicable to online data. Our theoretical and empirical results show that Local SGD simultaneously achieves both statistical efficiency and communication efficiency.

References in corpus (19)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Optimization in Heterogeneous Networks
• Sparsified SGD with Memory
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Local SGD with Periodic Averaging: Tighter Analysis and Adaptive Synchronization
• Variance Reduced Local SGD with Lower Communication Complexity
• FedSplit: An algorithmic framework for fast federated optimization
• First-order Newton-type Estimator for Distributed Estimation and Inference
• HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
• On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration
• Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling
• Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradients
• The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication
• Statistical inference using SGD
• Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLT

Cited by in corpus (1)

• Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling