Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data

Abstract

Federated learning allows loads of edge computing devices to collaboratively learn a global model without data sharing. The analysis with partial device participation under non-IID and unbalanced data reflects more reality. In this work, we propose federated learning versions of adaptive gradient methods - Federated AGMs - which employ both the first-order and second-order momenta, to alleviate generalization performance deterioration caused by dissimilarity of data population among devices. To further improve the test performance, we compare several schemes of calibration for the adaptive learning rate, including the standard Adam calibrated by $\epsilon$, $p$-Adam, and one calibrated by an activation function. Our analysis provides the first set of theoretical results that the proposed (calibrated) Federated AGMs converge to a first-order stationary point under non-IID and unbalanced data settings for nonconvex optimization. We perform extensive experiments to compare these federated learning methods with the state-of-the-art FedAvg, FedMomentum and SCAFFOLD and to assess the different calibration schemes and the advantages of AGMs over the current federated learning methods.

arXiv comment42 pages

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• ADADELTA: An Adaptive Learning Rate Method
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• On the Convergence of Adam and Beyond
• On the Convergence of FedAvg on Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Optimization in Heterogeneous Networks
• Adaptive Federated Optimization
• A Lyapunov Analysis of Momentum Methods in Optimization
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Faster On-Device Training Using New Federated Momentum Algorithm
• On the Outsized Importance of Learning Rates in Local Update Methods
• Stagewise Training Accelerates Convergence of Testing Error Over SGD
• Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions

Cited by in corpus (3)

• FedCM: Federated Learning with Client-level Momentum
• Local Adaptivity in Federated Learning: Convergence and Consistency
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data