AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training

Abstract

Highly distributed training of Deep Neural Networks (DNNs) on future compute platforms (offering 100 of TeraOps/s of computational capacity) is expected to be severely communication constrained. To overcome this limitation, new gradient compression techniques are needed that are computationally friendly, applicable to a wide variety of layers seen in Deep Neural Networks and adaptable to variations in network architectures as well as their hyper-parameters. In this paper we introduce a novel technique - the Adaptive Residual Gradient Compression (AdaComp) scheme. AdaComp is based on localized selection of gradient residues and automatically tunes the compression rate depending on local activity. We show excellent results on a wide spectrum of state of the art Deep Learning models in multiple domains (vision, speech, language), datasets (MNIST, CIFAR10, ImageNet, BN50, Shakespeare), optimizers (SGD with momentum, Adam) and network parameters (number of learners, minibatch-size etc.). Exploiting both sparsity and quantization, we demonstrate end-to-end compression rates of ~200X for fully-connected and recurrent layers, and ~40X for convolutional layers, without any noticeable degradation in model accuracies.

IBM Research AI, 9 pages, 7 figures, AAAI18 accepted

Cited by in corpus (29)

• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Communication optimization strategies for distributed deep neural network training: A survey
• Understanding Top-k Sparsification in Distributed Deep Learning
• Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Hyper-Sphere Quantization: Communication-Efficient SGD for Federated Learning
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• Communication-Efficient Edge AI: Algorithms and Systems
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning
• Communication-Efficient Decentralized Learning with Sparsification and Adaptive Peer Selection
• DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling
• Characterizing Impacts of Heterogeneity in Federated Learning upon Large-Scale Smartphone Data
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• A Distributed Synchronous SGD Algorithm with Global Top-$k$ Sparsification for Low Bandwidth Networks
• Revealing and Protecting Labels in Distributed Training
• Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach
• Fairness-aware Agnostic Federated Learning
• The Impact of GPU DVFS on the Energy and Performance of Deep Learning: an Empirical Study
• Is Network the Bottleneck of Distributed Training?
• Intermittent Pulling with Local Compensation for Communication-Efficient Federated Learning
• Distributed Deep Learning Strategies For Automatic Speech Recognition
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
• A Distributed SGD Algorithm with Global Sketching for Deep Learning Training Acceleration
• Gradient Sparification for Asynchronous Distributed Training
• Associative Convolutional Layers
• MergeComp: A Compression Scheduler for Scalable Communication-Efficient Distributed Training