Federated Optimization:Distributed Optimization Beyond the Datacenter

Abstract

We introduce a new and increasingly relevant setting for distributed optimization in machine learning, where the data defining the optimization are distributed (unevenly) over an extremely large number of \nodes, but the goal remains to train a high-quality centralized model. We refer to this setting as Federated Optimization. In this setting, communication efficiency is of utmost importance. A motivating example for federated optimization arises when we keep the training data locally on users' mobile devices rather than logging it to a data center for training. Instead, the mobile devices are used as nodes performing computation on their local data in order to update a global model. We suppose that we have an extremely large number of devices in our network, each of which has only a tiny fraction of data available totally; in particular, we expect the number of data points available locally to be much smaller than the number of devices. Additionally, since different users generate data with different patterns, we assume that no device has a representative sample of the overall distribution. We show that existing algorithms are not suitable for this setting, and propose a new algorithm which shows encouraging experimental results. This work also sets a path for future research needed in the context of federated optimization.

NIPS workshop version

References in corpus (1)

• Communication-Efficient Distributed Dual Coordinate Ascent

Cited by in corpus (129)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Federated Learning: Strategies for Improving Communication Efficiency
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• On the Convergence of FedAvg on Non-IID Data
• Edge Learning for B5G Networks with Distributed Signal Processing: Semantic Communication, Edge Computing, and Wireless Sensing
• Challenges in Deploying Machine Learning: a Survey of Case Studies
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Communication-Efficient Federated Deep Learning with Asynchronous Model Update and Temporally Weighted Aggregation
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Federated Learning with Personalization Layers
• Asynchronous Federated Optimization
• Precision Health Data: Requirements, Challenges and Existing Techniques for Data Security and Privacy
• Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review
• Don't Use Large Mini-Batches, Use Local SGD
• Inverting Gradients -- How easy is it to break privacy in federated learning?
• Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
• Decentralized Federated Learning: A Segmented Gossip Approach
• A Decentralized Federated Learning Framework via Committee Mechanism with Convergence Guarantee
• Edge Intelligence: Architectures, Challenges, and Applications
• Ground-Assisted Federated Learning in LEO Satellite Constellations
• Federated Multi-Task Learning
• AIDE: Fast and Communication Efficient Distributed Optimization
• A Joint Learning and Communications Framework for Federated Learning over Wireless Networks
• Federated Deep Reinforcement Learning
• Concept drift detection and adaptation for federated and continual learning
• Continual Local Training for Better Initialization of Federated Models
• Federated Learning for Healthcare Informatics
• Central Server Free Federated Learning over Single-sided Trust Social Networks
• Private Federated Learning with Domain Adaptation
• A first look into the carbon footprint of federated learning
• Communication-Efficient Federated Learning with Binary Neural Networks
• Throughput-Optimal Topology Design for Cross-Silo Federated Learning
• Federated Learning with Unbiased Gradient Aggregation and Controllable Meta Updating
• Federated PCA on Grassmann Manifold for IoT Anomaly Detection
• Stochastic, Distributed and Federated Optimization for Machine Learning
• Federated Learning With Quantized Global Model Updates
• Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs
• FedVQCS: Federated Learning via Vector Quantized Compressed Sensing
• SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks
• Communication-Efficient Local Decentralized SGD Methods
• Mobile Edge Intelligence and Computing for the Internet of Vehicles
• Not one but many Tradeoffs: Privacy Vs. Utility in Differentially Private Machine Learning
• How to Put Users in Control of their Data in Federated Top-N Recommendation with Learning to Rank
• Differentially Private Federated Clustering over Non-IID Data
• Communication-Efficient Edge AI: Algorithms and Systems
• Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
• Federated Dynamic GNN with Secure Aggregation
• FEDZIP: A Compression Framework for Communication-Efficient Federated Learning
• Fast Federated Learning in the Presence of Arbitrary Device Unavailability
• FedGiA: An Efficient Hybrid Algorithm for Federated Learning
• A Federated Learning Approach for Mobile Packet Classification
• Optimal Algorithms for Distributed Optimization
• BAFFLE : Blockchain Based Aggregator Free Federated Learning
• FedDICE: A ransomware spread detection in a distributed integrated clinical environment using federated learning and SDN based mitigation
• Adaptive and Parallel Split Federated Learning in Vehicular Edge Computing
• KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Survey
• Efficient, high-performance pancreatic segmentation using multi-scale feature extraction
• Preserving Patient Privacy while Training a Predictive Model of In-hospital Mortality
• Communication trade-offs for synchronized distributed SGD with large step size
• Communication-Efficient ADMM-based Federated Learning
• Distributed Stochastic Multi-Task Learning with Graph Regularization
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Less than a Single Pass: Stochastically Controlled Stochastic Gradient Method
• COVID-19 Imaging Data Privacy by Federated Learning Design: A Theoretical Framework
• Distributed Momentum for Byzantine-resilient Learning
• Data Poisoning Attacks on Federated Machine Learning
• LoAdaBoost: loss-based AdaBoost federated machine learning with reduced computational complexity on IID and non-IID intensive care data
• Bias-Variance Reduced Local SGD for Less Heterogeneous Federated Learning
• Communication-Efficient Robust Federated Learning Over Heterogeneous Datasets
• Ternary Compression for Communication-Efficient Federated Learning
• Efficient Learning of Interpretable Classification Rules
• Network Support for High-performance Distributed Machine Learning
• Distributed Stochastic Gradient Descent: Nonconvexity, Nonsmoothness, and Convergence to Local Minima
• Private Learning on Networks: Part II
• Optimal query complexity for private sequential learning against eavesdropping
• SoK: Machine Learning Governance
• Federated Composite Optimization
• Communication-Efficient Distributed Optimization in Networks with Gradient Tracking and Variance Reduction
• Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
• Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspective
• FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning
• MGN-Net: a multi-view graph normalizer for integrating heterogeneous biological network populations
• On Primal-Dual Approach for Distributed Stochastic Convex Optimization over Networks
• Exploring privacy-enhancing technologies in the automotive value chain
• Stochastic Channel-Based Federated Learning for Medical Data Privacy Preserving
• Toward an Automated Auction Framework for Wireless Federated Learning Services Market
• Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning
• Tackling Dynamics in Federated Incremental Learning with Variational Embedding Rehearsal
• Federated Transfer Learning with Dynamic Gradient Aggregation
• Federated Echo State Learning for Minimizing Breaks in Presence in Wireless Virtual Reality Networks
• Gradient Perturbation is Underrated for Differentially Private Convex Optimization
• Resource Scheduling in Edge Computing: A Survey
• ESMFL: Efficient and Secure Models for Federated Learning
• A Unified Transferable Model for ML-Enhanced DBMS
• Privacy Preserving Stochastic Channel-Based Federated Learning with Neural Network Pruning
• Fast Analog Transmission for High-Mobility Wireless Data Acquisition in Edge Learning
• AVDDPG: Federated reinforcement learning applied to autonomous platoon control
• Differentially Private Distributed Computation via Public-Private Communication Networks
• SLSGD: Secure and Efficient Distributed On-device Machine Learning
• A Compressive Sensing Approach for Federated Learning over Massive MIMO Communication Systems
• Distributed Linear Model Clustering over Networks: A Tree-Based Fused-Lasso ADMM Approach
• Federated Multi-task Hierarchical Attention Model for Sensor Analytics
• Distributed Machine Learning for Wireless Communication Networks: Techniques, Architectures, and Applications
• Differentially Private Federated Learning via Inexact ADMM
• A Communication-efficient Local Differentially Private Algorithm in Federated Optimization
• Exact Support Recovery in Federated Regression with One-shot Communication
• Strategyproof Learning: Building Trustworthy User-Generated Datasets
• Is In-Domain Data Really Needed? A Pilot Study on Cross-Domain Calibration for Network Quantization
• Utility Fairness for the Differentially Private Federated Learning
• Efficient Client Contribution Evaluation for Horizontal Federated Learning
• MixCon: Adjusting the Separability of Data Representations for Harder Data Recovery
• Optimising cost vs accuracy of decentralised analytics in fog computing environments
• New Directions in Distributed Deep Learning: Bringing the Network at Forefront of IoT Design
• Robust Federated Learning with Noisy Communication
• Federated Myopic Community Detection with One-shot Communication
• Distributed deep learning on edge-devices: feasibility via adaptive compression
• Test-time Collective Prediction
• Dynamic Gradient Aggregation for Federated Domain Adaptation
• Efficient Byzantine-Resilient Stochastic Gradient Desce
• A Privacy-Preserving and Trustable Multi-agent Learning Framework
• Federated Classification using Parsimonious Functions in Reproducing Kernel Hilbert Spaces
• Sketching Linear Classifiers over Data Streams
• Secure Distributed On-Device Learning Networks With Byzantine Adversaries
• Sequence-level self-learning with multiple hypotheses
• An Experiment Study on Federated LearningTestbed
• Towards More Efficient Federated Learning with Better Optimization Objects
• Canoe : A System for Collaborative Learning for Neural Nets
• Communication-Efficient Network-Distributed Optimization with Differential-Coded Compressors