A Distributed Hierarchical SGD Algorithm with Sparse Global Reduction

Abstract

Reducing communication in training large-scale machine learning applications on distributed platform is still a big challenge. To address this issue, we propose a distributed hierarchical averaging stochastic gradient descent (Hier-AVG) algorithm with infrequent global reduction by introducing local reduction. As a general type of parallel SGD, Hier-AVG can reproduce several popular synchronous parallel SGD variants by adjusting its parameters. We show that Hier-AVG with infrequent global reduction can still achieve standard convergence rate for non-convex optimization problems. In addition, we show that more frequent local averaging with more participants involved can lead to faster training convergence. By comparing Hier-AVG with another popular distributed training algorithm K-AVG, we show that through deploying local averaging with fewer number of global averaging, Hier-AVG can still achieve comparable training speed while frequently get better test accuracy. This indicates that local averaging can serve as an alternative remedy to effectively reduce communication overhead when the number of learners is large. Experimental results of Hier-AVG with several state-of-the-art deep neural nets on CIFAR-10 and IMAGENET-1K are presented to validate our analysis and show its superiority.

38 pages

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Revisiting Distributed Synchronous SGD
• Optimal Distributed Online Prediction using Mini-Batches
• Don't Use Large Mini-Batches, Use Local SGD
• Local SGD Converges Fast and Communicates Little
• Parallel SGD: When does averaging help?
• Memory and Communication Efficient Distributed Stochastic Optimization with Minibatch-Prox

Cited by in corpus (4)

• Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization
• Hierarchical Federated Learning with Quantization: Convergence Analysis and System Design
• Multi-Level Local SGD for Heterogeneous Hierarchical Networks
• A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to Balance Communication Overhead, Computational Complexity, and Convergence Rate