Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Abstract

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge and may severely deteriorate the generalization performance. In this paper, we investigate and identify the limitation of several decentralized optimization algorithms for different degrees of data heterogeneity. We propose a novel momentum-based method to mitigate this decentralized training difficulty. We show in extensive empirical experiments on various CV/NLP datasets (CIFAR-10, ImageNet, and AG News) and several network topologies (Ring and Social Network) that our method is much more robust to the heterogeneity of clients' data than other existing methods, by a significant improvement in test performance ($1\%-20\%$). Our code is publicly available.

References in corpus (37)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Federated Learning with Non-IID Data
• Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classification
• Optimal Distributed Online Prediction using Mini-Batches
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• FedML: A Research Library and Benchmark for Federated Machine Learning
• FedBN: Federated Learning on Non-IID Features via Local Batch Normalization
• Don't Use Large Mini-Batches, Use Local SGD
• HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• Communication Compression for Decentralized Training
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Bayesian Nonparametric Federated Learning of Neural Networks
• Stochastic Gradient Push for Distributed Deep Learning
• Variance-Reduced Decentralized Stochastic Optimization with Accelerated Convergence
• Norm matters: efficient and accurate normalization schemes in deep networks
• Mime: Mimicking Centralized Stochastic Algorithms in Federated Learning
• Quasi-hyperbolic momentum and Adam for deep learning
• On the Influence of Bias-Correction on Distributed Stochastic Optimization
• Towards Understanding Regularization in Batch Normalization
• Understanding the Role of Momentum in Stochastic Gradient Methods
• Distributed Stochastic Variance Reduced Gradient Methods and A Lower Bound for Communication Complexity
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• Quantized Decentralized Stochastic Learning over Directed Graphs
• Moniqua: Modulo Quantized Communication in Decentralized SGD
• Can Primal Methods Outperform Primal-dual Methods in Decentralized Dynamic Optimization?
• Consensus Control for Decentralized Deep Learning
• Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
• Faster Non-Convex Federated Learning via Global and Local Momentum
• Decentralized gradient methods: does topology matter?
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning

Cited by in corpus (10)

• Recent theoretical advances in decentralized distributed convex optimization
• FedCM: Federated Learning with Client-level Momentum
• Removing Data Heterogeneity Influence Enhances Network Topology Dependence of Decentralized SGD
• RelaySum for Decentralized Deep Learning on Heterogeneous Data
• Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Data
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• Stochastic Normalized Gradient Descent with Momentum for Large-Batch Training
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Multi-task Federated Edge Learning (MtFEEL) in Wireless Networks
• ${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning