Adaptive Federated Optimization

Abstract

Federated learning is a distributed machine learning paradigm in which a large number of clients coordinate with a central server to learn a model without sharing their own training data. Standard federated optimization methods such as Federated Averaging (FedAvg) are often difficult to tune and exhibit unfavorable convergence behavior. In non-federated settings, adaptive optimization methods have had notable success in combating such issues. In this work, we propose federated versions of adaptive optimizers, including Adagrad, Adam, and Yogi, and analyze their convergence in the presence of heterogeneous data for general non-convex settings. Our results highlight the interplay between client heterogeneity and communication efficiency. We also perform extensive experiments on these methods and show that the use of adaptive optimizers can significantly improve the performance of federated learning.

arXiv commentPublished as a conference paper at ICLR 2021

References in corpus (21)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• On the Convergence of Adam and Beyond
• One weird trick for parallelizing convolutional neural networks
• On the Convergence of FedAvg on Non-IID Data
• Towards Federated Learning at Scale: System Design
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Federated Optimization in Heterogeneous Networks
• LEAF: A Benchmark for Federated Settings
• How to Escape Saddle Points Efficiently
• Stochastic Variance Reduction for Nonconvex Optimization
• Adaptive Gradient Methods with Dynamic Bound of Learning Rate
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Adaptive Bound Optimization for Online Convex Optimization
• The Non-IID Data Quagmire of Decentralized Machine Learning
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• First Analysis of Local GD on Heterogeneous Data
• Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Network
• A Generic Approach for Escaping Saddle points
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

Cited by in corpus (81)

• A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protection
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• FedML: A Research Library and Benchmark for Federated Machine Learning
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Personalized Federated Learning with Moreau Envelopes
• Inverting Gradients -- How easy is it to break privacy in federated learning?
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Federated Learning Based on Dynamic Regularization
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Differentially Private Learning with Adaptive Clipping
• Federated Learning on Non-IID Data Silos: An Experimental Study
• Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategies
• Distilled One-Shot Federated Learning
• Training Speech Recognition Models with Federated Learning: A Quality/Cost Framework
• Exploiting Shared Representations for Personalized Federated Learning
• FedCluster: Boosting the Convergence of Federated Learning via Cluster-Cycling
• A first look into the carbon footprint of federated learning
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks
• FLASHE: Additively Symmetric Homomorphic Encryption for Cross-Silo Federated Learning
• FedScale: Benchmarking Model and System Performance of Federated Learning at Scale
• Federated Learning via Posterior Averaging: A New Perspective and Practical Algorithms
• The Skellam Mechanism for Differentially Private Federated Learning
• Federated Learning with Buffered Asynchronous Aggregation
• Papaya: Practical, Private, and Scalable Federated Learning
• SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Prototype Guided Federated Learning of Visual Feature Representations
• The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation
• Federated Reconstruction: Partially Local Federated Learning
• FedCM: Federated Learning with Client-level Momentum
• Practical and Private (Deep) Learning without Sampling or Shuffling
• Local Adaptivity in Federated Learning: Convergence and Consistency
• FedJAX: Federated learning simulation with JAX
• On the Outsized Importance of Learning Rates in Local Update Methods
• FedEval: A Holistic Evaluation Framework for Federated Learning
• Improving Semi-supervised Federated Learning by Reducing the Gradient Diversity of Models
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learning
• Secure Aggregation for Buffered Asynchronous Federated Learning
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• SSFL: Tackling Label Deficiency in Federated Learning via Personalized Self-Supervision
• FedNL: Making Newton-Type Methods Applicable to Federated Learning
• Efficient and Private Federated Learning with Partially Trainable Networks
• Federated Composite Optimization
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Federated Mixture of Experts
• RingFed: Reducing Communication Costs in Federated Learning on Non-IID Data
• Federated Learning on Non-IID Data: A Survey
• Federated Transfer Learning with Dynamic Gradient Aggregation
• Secure Byzantine-Robust Distributed Learning via Clustering
• Convergence and Accuracy Trade-Offs in Federated Learning and Meta-Learning
• Multi-VFL: A Vertical Federated Learning System for Multiple Data and Label Owners
• Anarchic Federated Learning
• Cost-Effective Federated Learning in Mobile Edge Networks
• IFedAvg: Interpretable Data-Interoperability for Federated Learning
• Federated Nonconvex Sparse Learning
• Federated Class Incremental Learning: A Pseudo Feature Based Approach Without Exemplars
• Adaptive Serverless Learning
• Policy-Based Federated Learning
• Towards Model Agnostic Federated Learning Using Knowledge Distillation
• Communication-Efficient Agnostic Federated Averaging
• DP-REC: Private & Communication-Efficient Federated Learning
• Dubhe: Towards Data Unbiasedness with Homomorphic Encryption in Federated Learning Client Selection
• ABC-FL: Anomalous and Benign client Classification in Federated Learning
• Adaptive Differentially Private Empirical Risk Minimization
• An Operator Splitting View of Federated Learning
• EasyFL: A Low-code Federated Learning Platform For Dummies
• Quantized Adam with Error Feedback
• Accelerating Federated Learning in Heterogeneous Data and Computational Environments
• CADA: Communication-Adaptive Distributed Adam
• An Expectation-Maximization Perspective on Federated Learning
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• Federated Unbiased Learning to Rank
• Dynamic Gradient Aggregation for Federated Domain Adaptation
• Compositional federated learning: Applications in distributionally robust averaging and meta learning
• Federated Learning with Sparsification-Amplified Privacy and Adaptive Optimization
• Design and Analysis of Uplink and Downlink Communications for Federated Learning