Consensus Control for Decentralized Deep Learning

Abstract

Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers from the degradation in the quality of the model: the training and test performance of models trained in a decentralized fashion is in general worse than that of models trained in a centralized fashion, and this performance drop is impacted by parameters such as network size, communication topology and data partitioning. We identify the changing consensus distance between devices as a key parameter to explain the gap between centralized and decentralized training. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. We empirically validate that the relation between generalization performance and consensus distance is consistent with this theoretical observation. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop. To this end, we provide practical training guidelines and exemplify its effectiveness on the data-center setup as the important first step.

arXiv commentLK and TL contribute equally - ICML 2021

References in corpus (26)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Optimal Distributed Online Prediction using Mini-Batches
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Averaging Weights Leads to Wider Optima and Better Generalization
• Don't Use Large Mini-Batches, Use Local SGD
• Optimal algorithms for smooth and strongly convex distributed optimization in networks
• Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication
• A Unified Theory of Decentralized SGD with Changing Topology and Local Updates
• Measuring the Effects of Data Parallelism on Neural Network Training
• Stochastic Gradient Push for Distributed Deep Learning
• Optimal Algorithms for Non-Smooth Distributed Optimization in Networks
• Implicit Regularization in Deep Learning
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Distributed Non-Convex First-Order Optimization and Information Processing: Lower Complexity Bounds and Rate Optimal Algorithms
• Unified Optimal Analysis of the (Stochastic) Gradient Method
• On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length
• The Early Phase of Neural Network Training
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• AdaScale SGD: A User-Friendly Algorithm for Distributed Training
• Emergent properties of the local geometry of neural loss landscapes
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Decentralized gradient methods: does topology matter?
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• On Consensus-Optimality Trade-offs in Collaborative Deep Learning
• Extrapolation for Large-batch Training in Deep Learning

Cited by in corpus (6)

• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• Decentralized federated learning of deep neural networks on non-iid data
• BlueFog: Make Decentralized Algorithms Practical for Optimization and Deep Learning
• Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Data
• ${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications