Cross-Gradient Aggregation for Decentralized Learning from Non-IID data

Abstract

Decentralized learning enables a group of collaborative agents to learn models using a distributed dataset without the need for a central parameter server. Recently, decentralized learning algorithms have demonstrated state-of-the-art results on benchmark data sets, comparable with centralized algorithms. However, the key assumption to achieve competitive performance is that the data is independently and identically distributed (IID) among the agents which, in real-life applications, is often not applicable. Inspired by ideas from continual learning, we propose Cross-Gradient Aggregation (CGA), a novel decentralized learning algorithm where (i) each agent aggregates cross-gradient information, i.e., derivatives of its model with respect to its neighbors' datasets, and (ii) updates its model using a projected gradient based on quadratic programming (QP). We theoretically analyze the convergence characteristics of CGA and demonstrate its efficiency on non-IID data distributions sampled from the MNIST and CIFAR-10 datasets. Our empirical comparisons show superior learning performance of CGA over existing state-of-the-art decentralized learning algorithms, as well as maintaining the improved performance under information compression to reduce peer-to-peer communication overhead. The code is available here on GitHub.

ICML 2021 accepted paper

References in corpus (22)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Federated Learning with Non-IID Data
• Federated Optimization: Distributed Machine Learning for On-Device Intelligence
• On the Convergence of FedAvg on Non-IID Data
• SCAFFOLD: Stochastic Controlled Averaging for Federated Learning
• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Federated Optimization in Heterogeneous Networks
• Distributed Coordinate Descent Method for Learning with Big Data
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization
• Overcoming Forgetting in Federated Learning on Non-IID Data
• Stochastic Gradient Push for Distributed Deep Learning
• DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
• The Non-IID Data Quagmire of Decentralized Machine Learning
• SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• Moniqua: Modulo Quantized Communication in Decentralized SGD
• Decentralized Deep Learning with Arbitrary Communication Compression
• Effective Federated Adaptive Gradient Methods with Non-IID Decentralized Data
• IDEAL: Inexact DEcentralized Accelerated Augmented Lagrangian Method
• Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
• On Consensus-Optimality Trade-offs in Collaborative Deep Learning