Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating

Abstract

Federated learning (FL) aims to train machine learning models in the decentralized system consisting of an enormous amount of smart edge devices. Federated averaging (FedAvg), the fundamental algorithm in FL settings, proposes on-device training and model aggregation to avoid the potential heavy communication costs and privacy concerns brought by transmitting raw data. However, through theoretical analysis we argue that 1) the multiple steps of local updating will result in gradient biases and 2) there is an inconsistency between the expected target distribution and the optimization objectives following the training paradigm in FedAvg. To tackle these problems, we first propose an unbiased gradient aggregation algorithm with the keep-trace gradient descent and the gradient evaluation strategy. Then we introduce an additional controllable meta updating procedure with a small set of data samples, indicating the expected target distribution, to provide a clear and consistent optimization objective. Both the two improvements are model- and task-agnostic and can be applied individually or together. Experimental results demonstrate that the proposed methods are faster in convergence and achieve higher accuracy with different network architectures in various FL settings.

arXiv commentThis manuscript has been accepted to the Workshop on Federated Learning for Data Privacy and Confidentiality (FL - NeurIPS 2019, in Conjunction with NeurIPS 2019)

References in corpus (22)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• On the Convergence of FedAvg on Non-IID Data
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Secure Federated Transfer Learning
• Federated Optimization:Distributed Optimization Beyond the Datacenter
• EMNIST: an extension of MNIST to handwritten letters
• Federated Optimization in Heterogeneous Networks
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Improving Federated Learning Personalization via Model Agnostic Meta Learning
• Personalized Federated Learning: A Meta-Learning Approach
• LEAF: A Benchmark for Federated Settings
• Agnostic Federated Learning
• SecureBoost: A Lossless Federated Learning Framework
• Expanding the Reach of Federated Learning by Reducing Client Resource Requirements
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• Robust Federated Training via Collaborative Machine Teaching using Trusted Instances

Cited by in corpus (10)

• A Framework for Evaluating Gradient Leakage Attacks in Federated Learning
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Turn Signal Prediction: A Federated Learning Case Study
• Aggregate or Not? Exploring Where to Privatize in DNN Based Federated Learning Under Different Non-IID Scenes
• ByGARS: Byzantine SGD with Arbitrary Number of Attackers
• Bias-Free FedGAN: A Federated Approach to Generate Bias-Free Datasets
• SPOCC: Scalable POssibilistic Classifier Combination -- toward robust aggregation of classifiers
• Gradient-Leakage Resilient Federated Learning
• Gradient Masked Federated Optimization