Federated Learning Based on Dynamic Regularization

Abstract

We propose a novel federated learning method for distributively training neural network models, where the server orchestrates cooperation between a subset of randomly chosen devices in each round. We view Federated Learning problem primarily from a communication perspective and allow more device level computations to save transmission costs. We point out a fundamental dilemma, in that the minima of the local-device level empirical loss are inconsistent with those of the global empirical loss. Different from recent prior works, that either attempt inexact minimization or utilize devices for parallelizing gradient computation, we propose a dynamic regularizer for each device at each round, so that in the limit the global and device solutions are aligned. We demonstrate both through empirical results on real and synthetic data as well as analytical results that our scheme leads to efficient training, in both convex and non-convex settings, while being fully agnostic to device heterogeneity and robust to large number of devices, partial participation and unbalanced data.

arXiv commentSlightly extended version of ICLR 2021 Paper

References in corpus (13)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• LEAF: A Benchmark for Federated Settings
• Bayesian Nonparametric Federated Learning of Neural Networks
• Adaptive Federated Optimization
• Distributed Learning with Compressed Gradient Differences
• Variance Reduced Local SGD with Lower Communication Complexity
• FedSplit: An algorithmic framework for fast federated optimization
• A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
• Distributed Proximal Splitting Algorithms with Rates and Acceleration

Cited by in corpus (24)

• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data
• A Field Guide to Federated Optimization
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Federated Learning on Non-IID Data Silos: An Experimental Study
• FedLab: A Flexible Federated Learning Framework
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Differentially Private Federated Learning on Heterogeneous Data
• Speeding up Heterogeneous Federated Learning with Sequentially Trained Superclients
• Model-Contrastive Federated Learning
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• Federated Hyperparameter Tuning: Challenges, Baselines, and Connections to Weight-Sharing
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• Anarchic Federated Learning
• Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification
• Communication-Efficient Agnostic Federated Averaging
• FedProf: Selective Federated Learning with Representation Profiling
• FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Distantly Supervised Relation Extraction in Federated Settings
• Accelerating Federated Learning by Selecting Beneficial Herd of Local Gradients