Faster On-Device Training Using New Federated Momentum Algorithm

Abstract

Mobile crowdsensing has gained significant attention in recent years and has become a critical paradigm for emerging Internet of Things applications. The sensing devices continuously generate a significant quantity of data, which provide tremendous opportunities to develop innovative intelligent applications. To utilize these data to train machine learning models while not compromising user privacy, federated learning has become a promising solution. However, there is little understanding of whether federated learning algorithms are guaranteed to converge. We reconsider model averaging in federated learning and formulate it as a gradient-based method with biased gradients. This novel perspective assists analysis of its convergence rate and provides a new direction for more acceleration. We prove for the first time that the federated averaging algorithm is guaranteed to converge for non-convex problems, without imposing additional assumptions. We further propose a novel accelerated federated learning algorithm and provide a convergence guarantee. Simulated federated learning experiments are conducted to train deep neural networks on benchmark datasets, and experimental results show that our proposed method converges faster than previous approaches.

References in corpus (20)

• Decoupled Weight Decay Regularization
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning for Mobile Keyboard Prediction
• Character-Aware Neural Language Models
• On the Convergence of FedAvg on Non-IID Data
• Towards Federated Learning at Scale: System Design
• Learning Differentially Private Recurrent Language Models
• Revisiting Distributed Synchronous SGD
• Large Batch Training of Convolutional Networks
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• LEAF: A Benchmark for Federated Settings
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• SecureBoost: A Lossless Federated Learning Framework
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization
• Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity

Cited by in corpus (14)

• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Federated Learning with Nesterov Accelerated Gradient
• Federated Learning With Quantized Global Model Updates
• Federated Learning Meets Multi-objective Optimization
• FedCM: Federated Learning with Client-level Momentum
• Federated Learning for Energy Constrained IoT devices: A systematic mapping study
• Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement
• Federated Optimization of Smooth Loss Functions
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Communication-Efficient Federated Learning with Compensated Overlap-FedAvg
• An Operator Splitting View of Federated Learning