GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent

Abstract

In this paper, we present GossipGraD - a gossip communication protocol based Stochastic Gradient Descent (SGD) algorithm for scaling Deep Learning (DL) algorithms on large-scale systems. The salient features of GossipGraD are: 1) reduction in overall communication complexity from Θ(log(p)) for p compute nodes in well-studied SGD to O(1), 2) model diffusion such that compute nodes exchange their updates (gradients) indirectly after every log(p) steps, 3) rotation of communication partners for facilitating direct diffusion of gradients, 4) asynchronous distributed shuffle of samples during the feedforward phase in SGD to prevent over-fitting, 5) asynchronous communication of gradients for further reducing the communication cost of SGD and GossipGraD. We implement GossipGraD for GPU and CPU clusters and use NVIDIA GPUs (Pascal P100) connected with InfiniBand, and Intel Knights Landing (KNL) connected with Aries network. We evaluate GossipGraD using well-studied dataset ImageNet-1K (~250GB), and widely studied neural network topologies such as GoogLeNet and ResNet50 (current winner of ImageNet Large Scale Visualization Research Challenge (ILSVRC)). Our performance evaluation using both KNL and Pascal GPUs indicates that GossipGraD can achieve perfect efficiency for these datasets and their associated neural network topologies. Specifically, for ResNet50, GossipGraD is able to achieve ~100% compute efficiency using 128 NVIDIA Pascal P100 GPUs - while matching the top-1 classification accuracy published in literature.

13 pages, 17 figures

References in corpus (18)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Going Deeper with Convolutions
• Theano: new features and speed improvements
• One weird trick for parallelizing convolutional neural networks
• Rethinking the Inception Architecture for Computer Vision
• Large Batch Training of Convolutional Networks
• Deep Image: Scaling up Image Recognition
• Application of Deep Convolutional Neural Networks for Detecting Extreme Weather in Climate Datasets
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• SparkNet: Training Deep Networks in Spark
• How to scale distributed deep learning?
• ImageNet Training in Minutes
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• Gossip training for deep learning
• Distributed TensorFlow with MPI

Cited by in corpus (20)

• Edge Intelligence: The Confluence of Edge Computing and Artificial Intelligence
• Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Opportunities of Federated Learning in Connected, Cooperative and Automated Industrial Systems
• Decentralized Federated Learning: A Segmented Gossip Approach
• Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
• Priority-based Parameter Propagation for Distributed DNN Training
• Communication-Efficient Edge AI: Algorithms and Systems
• Privacy Preservation in Federated Learning: An insightful survey from the GDPR Perspective
• NetReduce: RDMA-Compatible In-Network Reduction for Distributed DNN Training Acceleration
• EventGraD: Event-Triggered Communication in Parallel Machine Learning
• FLAGS Framework for Comparative Analysis of Federated Learning Algorithms
• On the Limit Performance of Floating Gossip
• State-of-the-art Techniques in Deep Edge Intelligence
• Crossover-SGD: A gossip-based communication in distributed deep learning for alleviating large mini-batch problem and enhancing scalability
• Detection of Insider Attacks in Distributed Projected Subgradient Algorithms
• Optimising cost vs accuracy of decentralised analytics in fog computing environments
• ${\text{A}}^2{\text{CiD}}^2$: Accelerating Asynchronous Communication in Decentralized Deep Learning