Federated Optimization in Heterogeneous Networks

Abstract

Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the network (statistical heterogeneity). In this work, we introduce a framework, FedProx, to tackle heterogeneity in federated networks. FedProx can be viewed as a generalization and re-parametrization of FedAvg, the current state-of-the-art method for federated learning. While this re-parameterization makes only minor modifications to the method itself, these modifications have important ramifications both in theory and in practice. Theoretically, we provide convergence guarantees for our framework when learning over data from non-identical distributions (statistical heterogeneity), and while adhering to device-level systems constraints by allowing each participating device to perform a variable amount of work (systems heterogeneity). Practically, we demonstrate that FedProx allows for more robust convergence than FedAvg across a suite of realistic federated datasets. In particular, in highly heterogeneous settings, FedProx demonstrates significantly more stable and accurate convergence behavior relative to FedAvg---improving absolute test accuracy by 22% on average.

MLSys 2020

References in corpus (17)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Federated Learning with Non-IID Data
• Towards Federated Learning at Scale: System Design
• Optimal Distributed Online Prediction using Mini-Batches
• EMNIST: an extension of MNIST to handwritten letters
• Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• LEAF: A Benchmark for Federated Settings
• Don't Use Large Mini-Batches, Use Local SGD
• Distributed Coordinate Descent Method for Learning with Big Data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Communication-Efficient Distributed Dual Coordinate Ascent
• AIDE: Fast and Communication Efficient Distributed Optimization
• Fast Convergence of Stochastic Gradient Descent under a Strong Growth Condition
• Graph Oracle Models, Lower Bounds, and Gaps for Parallel Stochastic Optimization
• How To Make the Gradients Small Stochastically: Even Faster Convex and Nonconvex SGD

Cited by in corpus (188)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• On the Convergence of FedAvg on Non-IID Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Privacy-preserving Traffic Flow Prediction: A Federated Learning Approach
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedMD: Heterogenous Federated Learning via Model Distillation
• FedML: A Research Library and Benchmark for Federated Machine Learning
• Adaptive Personalized Federated Learning
• A Crowdsourcing Framework for On-Device Federated Learning
• Federated Meta-Learning with Fast Convergence and Efficient Communication
• Personalized Federated Learning with Moreau Envelopes
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Learning to Detect Malicious Clients for Robust Federated Learning
• SecureBoost: A Lossless Federated Learning Framework
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Adaptive Federated Optimization
• A Federated Learning Aggregation Algorithm for Pervasive Computing: Evaluation and Comparison
• Variational Federated Multi-Task Learning
• Abnormal Client Behavior Detection in Federated Learning
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Concept drift detection and adaptation for federated and continual learning
• Distilled One-Shot Federated Learning
• Federated Mutual Learning
• Exploiting Shared Representations for Personalized Federated Learning
• FedSAE: A Novel Self-Adaptive Federated Learning Framework in Heterogeneous Systems
• Federated Learning for Healthcare Informatics
• First Analysis of Local GD on Heterogeneous Data
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Lower Bounds and Optimal Algorithms for Personalized Federated Learning
• A Communication Efficient Collaborative Learning Framework for Distributed Features
• A first look into the carbon footprint of federated learning
• Communication Efficiency in Federated Learning: Achievements and Challenges
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• FedBABU: Towards Enhanced Representation for Federated Image Classification
• Federated Bayesian Optimization via Thompson Sampling
• PPFL: Privacy-preserving Federated Learning with Trusted Execution Environments
• Privacy for Free: Communication-Efficient Learning with Differential Privacy Using Sketches
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data
• Tighter Theory for Local SGD on Identical and Heterogeneous Data
• Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
• Federated Learning with Buffered Asynchronous Aggregation
• Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentation
• SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training
• Federated User Representation Learning
• Faster On-Device Training Using New Federated Momentum Algorithm
• Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Data
• Personalized Federated Learning using Hypernetworks
• Convergence Time Optimization for Federated Learning over Wireless Networks
• Towards Flexible Device Participation in Federated Learning
• Federated Continual Learning with Weighted Inter-client Transfer
• Hierarchical Federated Learning through LAN-WAN Orchestration
• Federated Learning Meets Multi-objective Optimization
• FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Communication-Efficient Local Decentralized SGD Methods
• TornadoAggregate: Accurate and Scalable Federated Learning via the Ring-Based Architecture
• Communication-Efficient Edge AI: Algorithms and Systems
• Client-Edge-Cloud Hierarchical Federated Learning
• A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
• Federated Reconstruction: Partially Local Federated Learning
• FedCM: Federated Learning with Client-level Momentum
• GFL: A Decentralized Federated Learning Framework Based On Blockchain
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Robust Federated Learning Through Representation Matching and Adaptive Hyper-parameters
• STEM: A Stochastic Two-Sided Momentum Algorithm Achieving Near-Optimal Sample and Communication Complexities for Federated Learning
• FedJAX: Federated learning simulation with JAX
• Federated Residual Learning
• Training Fair Models in Federated Learning without Data Privacy Infringement
• Stragglers Are Not Disaster: A Hybrid Federated Learning Algorithm with Delayed Gradients
• Asynchronous Online Federated Learning for Edge Devices with Non-IID Data
• Asynchronous Federated Learning with Reduced Number of Rounds and with Differential Privacy from Less Aggregated Gaussian Noise
• Implicit Model Specialization through DAG-based Decentralized Federated Learning
• Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing
• Adversarial training in communication constrained federated learning
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspective
• Accurate and Fast Federated Learning via IID and Communication-Aware Grouping
• A Theoretical Perspective on Differentially Private Federated Multi-task Learning
• Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
• Fast-adapting and Privacy-preserving Federated Recommender System
• FedLoc: Federated Learning Framework for Data-Driven Cooperative Localization and Location Data Processing
• HFEL: Joint Edge Association and Resource Allocation for Cost-Efficient Hierarchical Federated Edge Learning
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• Exploiting Unlabeled Data in Smart Cities using Federated Learning
• Multi-institutional Collaborations for Improving Deep Learning-based Magnetic Resonance Image Reconstruction Using Federated Learning
• Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• FedNL: Making Newton-Type Methods Applicable to Federated Learning
• Applications of Federated Learning in Smart Cities: Recent Advances, Taxonomy, and Open Challenges
• Personalized Federated Learning with Gaussian Processes
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update
• FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Local SGD: Unified Theory and New Efficient Methods
• From Federated Learning to Federated Neural Architecture Search: A Survey
• Federated Mixture of Experts
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• Federated $f$-Differential Privacy
• RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
• Auction Based Clustered Federated Learning in Mobile Edge Computing System
• Federated Learning on Non-IID Data: A Survey
• Fast-Convergent Federated Learning with Adaptive Weighting
• FedPAGE: A Fast Local Stochastic Gradient Method for Communication-Efficient Federated Learning
• FedFm: Towards a Robust Federated Learning Approach For Fault Mitigation at the Edge Nodes
• Towards Fair Federated Learning with Zero-Shot Data Augmentation
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• Federated Block Coordinate Descent Scheme for Learning Global and Personalized Models
• Bandwidth Allocation for Multiple Federated Learning Services in Wireless Edge Networks
• Voting-based Approaches For Differentially Private Federated Learning
• Anarchic Federated Learning
• Fast Convergence Algorithm for Analog Federated Learning
• Federated Nonconvex Sparse Learning
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• FedHealth 2: Weighted Federated Transfer Learning via Batch Normalization for Personalized Healthcare
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Federated Learning with Positive and Unlabeled Data
• Accurate and Fast Federated Learning via Combinatorial Multi-Armed Bandits
• Prospects of federated machine learning in fluid dynamics
• Alternating Direction Method of Multipliers for Quantization
• Provably Efficient Cooperative Multi-Agent Reinforcement Learning with Function Approximation
• Server Averaging for Federated Learning
• Improving Federated Relational Data Modeling via Basis Alignment and Weight Penalty
• Learning Federated Representations and Recommendations with Limited Negatives
• Trends and Advancements in Deep Neural Network Communication
• FedSmart: An Auto Updating Federated Learning Optimization Mechanism
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data
• Communication-Efficient Agnostic Federated Averaging
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data
• Convergence Analysis and System Design for Federated Learning over Wireless Networks
• Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates
• Federated Learning for Open Banking
• Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection
• FedPrune: Towards Inclusive Federated Learning
• Data Selection for Efficient Model Update in Federated Learning
• Federated Learning with Fair Averaging
• Performance Analysis and Characterization of Training Deep Learning Models on Mobile Devices
• Federated Multiple Label Hashing (FedMLH): Communication Efficient Federated Learning on Extreme Classification Tasks
• Delayed Projection Techniques for Linearly Constrained Problems: Convergence Rates, Acceleration, and Applications
• Algorithm Fairness in AI for Medicine and Healthcare
• Tackling the Local Bias in Federated Graph Learning
• Local SGD Optimizes Overparameterized Neural Networks in Polynomial Time
• TiFL: A Tier-based Federated Learning System
• FedDropoutAvg: Generalizable federated learning for histopathology image classification
• Representation of Federated Learning via Worst-Case Robust Optimization Theory
• Towards Bidirectional Protection in Federated Learning
• Differentially Private Federated Learning via Inexact ADMM
• Hogwild! over Distributed Local Data Sets with Linearly Increasing Mini-Batch Sizes
• Accelerating Federated Learning in Heterogeneous Data and Computational Environments
• Personalized Federated Learning over non-IID Data for Indoor Localization
• Distributed Learning on Heterogeneous Resource-Constrained Devices
• Reliability and Performance Assessment of Federated Learning on Clinical Benchmark Data
• Asynchronous Federated Learning for Sensor Data with Concept Drift
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• Privacy-Preserving Constrained Domain Generalization via Gradient Alignment
• Adaptive Federated Learning With Gradient Compression in Uplink NOMA
• Federated Learning using Smart Contracts on Blockchains, based on Reward Driven Approach
• Statistical Estimation and Inference via Local SGD in Federated Learning
• Probabilistic Federated Learning of Neural Networks Incorporated with Global Posterior Information
• FedMM: Saddle Point Optimization for Federated Adversarial Domain Adaptation
• An Expectation-Maximization Perspective on Federated Learning
• Federated Dynamic Neural Network for Deep MIMO Detection
• Asynchronous Distributed Optimization with Stochastic Delays
• CADA: Communication-Adaptive Distributed Adam
• Differential Privacy Meets Federated Learning under Communication Constraints
• Demystifying the Effects of Non-Independence in Federated Learning
• Federated Few-Shot Learning with Adversarial Learning
• On-device Federated Learning with Flower
• Matrix Sketching for Secure Collaborative Machine Learning
• Loosely Coupled Federated Learning Over Generative Models
• Local SGD for Near-Quadratic Problems: Improving Convergence under Unconstrained Noise Conditions
• Gradient Masked Federated Optimization
• On Addressing Heterogeneity in Federated Learning for Autonomous Vehicles Connected to a Drone Orchestrator
• Sensing and Mapping for Better Roads: Initial Plan for Using Federated Learning and Implementing a Digital Twin to Identify the Road Conditions in a Developing Country -- Sri Lanka
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data
• Distantly Supervised Relation Extraction in Federated Settings
• Towards Heterogeneous Clients with Elastic Federated Learning
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Budgeted Online Selection of Candidate IoT Clients to Participate in Federated Learning
• A Personalized Federated Learning Algorithm: an Application in Anomaly Detection
• Can Federated Learning Save The Planet?
• Towards More Efficient Federated Learning with Better Optimization Objects