Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

Abstract

Large-scale distributed training requires significant communication bandwidth for gradient exchange that limits the scalability of multi-node training, and requires expensive high-bandwidth network infrastructure. The situation gets even worse with distributed training on mobile devices (federated learning), which suffers from higher latency, lower throughput, and intermittent poor connections. In this paper, we find 99.9% of the gradient exchange in distributed SGD is redundant, and propose Deep Gradient Compression (DGC) to greatly reduce the communication bandwidth. To preserve accuracy during compression, DGC employs four methods: momentum correction, local gradient clipping, momentum factor masking, and warm-up training. We have applied Deep Gradient Compression to image classification, speech recognition, and language modeling with multiple datasets including Cifar10, ImageNet, Penn Treebank, and Librispeech Corpus. On these scenarios, Deep Gradient Compression achieves a gradient compression ratio from 270x to 600x without losing accuracy, cutting the gradient size of ResNet-50 from 97MB to 0.35MB, and for DeepSpeech from 488MB to 0.74MB. Deep gradient compression enables large-scale distributed training on inexpensive commodity 1Gbps Ethernet and facilitates distributed training on mobile. Code is available at: https://github.com/synxlin/deep-gradient-compression.

arXiv commentwe find 99.9% of the gradient exchange in distributed SGD is redundant; we reduce the communication bandwidth by two orders of magnitude without losing accuracy. Code is available at: https://github.com/synxlin/deep-gradient-compression

References in corpus (10)

• Communication-Efficient Learning of Deep Networks from Decentralized Data
• On the difficulty of training Recurrent Neural Networks
• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• SparkNet: Training Deep Networks in Spark

Cited by in corpus (192)

• Federated Learning with Non-IID Data
• Convergence of Edge Computing and Deep Learning: A Comprehensive Survey
• Federated Machine Learning: Concept and Applications
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Vertical Federated Learning: Concepts, Advances and Challenges
• FedML: A Research Library and Benchmark for Federated Machine Learning
• A Crowdsourcing Framework for On-Device Federated Learning
• Multi-Armed Bandit Based Client Scheduling for Federated Learning
• UVeQFed: Universal Vector Quantization for Federated Learning
• Benchmarking TPU, GPU, and CPU Platforms for Deep Learning
• Don't Use Large Mini-Batches, Use Local SGD
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Local SGD Converges Fast and Communicates Little
• Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge
• Federated Learning: A Signal Processing Perspective
• Split learning for health: Distributed deep learning without sharing raw patient data
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Database Meets Deep Learning: Challenges and Opportunities
• Federated Learning in Mobile Edge Networks: A Comprehensive Survey
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Federated Learning for Internet of Things: A Federated Learning Framework for On-device Anomaly Data Detection
• Edge Intelligence: Architectures, Challenges, and Applications
• Scaling Distributed Machine Learning with In-Network Aggregation
• Federated Continual Learning via Knowledge Fusion: A Survey
• Robust Federated Learning with Noisy Labels
• GRNN: Generative Regression Neural Network -- A Data Leakage Attack for Federated Learning
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Parameter Hub: a Rack-Scale Parameter Server for Distributed Deep Neural Network Training
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• The Non-IID Data Quagmire of Decentralized Machine Learning
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform
• Fast Federated Learning by Balancing Communication Trade-Offs
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• Communication optimization strategies for distributed deep neural network training: A survey
• Joint Privacy Enhancement and Quantization in Federated Learning
• Natural Compression for Distributed Deep Learning
• FetchSGD: Communication-Efficient Federated Learning with Sketching
• FedLab: A Flexible Federated Learning Framework
• Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
• Computation Scheduling for Distributed Machine Learning with Straggling Workers
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• Towards an Intelligent Edge: Wireless Communication Meets Machine Learning
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Federated Learning via Over-the-Air Computation
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• On Biased Compression for Distributed Learning
• Data Shuffling in Wireless Distributed Computing via Low-Rank Optimization
• The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems
• Towards Communication-efficient and Attack-Resistant Federated Edge Learning for Industrial Internet of Things
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• A Reputation Mechanism Is All You Need: Collaborative Fairness and Adversarial Robustness in Federated Learning
• Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Backdoor attacks and defenses in feature-partitioned collaborative learning
• Optimal Gradient Compression for Distributed and Federated Learning
• A Federated Deep Learning Framework for Privacy Preservation and Communication Efficiency
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• Similarity-based Label Inference Attack against Training and Inference of Split Learning
• Federated Learning With Quantized Global Model Updates
• Edge-assisted Democratized Learning Towards Federated Analytics
• Decentralized Deep Learning with Arbitrary Communication Compression
• DEFER: Distributed Edge Inference for Deep Neural Networks
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• HashVFL: Defending Against Data Reconstruction Attacks in Vertical Federated Learning
• Communication-Efficient Local Decentralized SGD Methods
• Communication-Efficient Edge AI: Algorithms and Systems
• Recent Advances in Deep Learning for Object Detection
• Deep Leakage from Gradients
• Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
• Decentralized Learning of Generative Adversarial Networks from Non-iid Data
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
• Communication-Efficient Robust Federated Learning with Noisy Labels
• Towards Reliable Participation in UAV-Enabled Federated Edge Learning on Non-IID Data
• Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
• FedEval: A Holistic Evaluation Framework for Federated Learning
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• On the Utility of Gradient Compression in Distributed Training Systems
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• An On-Device Federated Learning Approach for Cooperative Model Update between Edge Devices
• Communication-efficient distributed SGD with Sketching
• Shielding Collaborative Learning: Mitigating Poisoning Attacks through Client-Side Detection
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• Distributed Momentum for Byzantine-resilient Learning
• Decentralized Learning Made Easy with DecentralizePy
• On the Convergence of SGD with Biased Gradients
• Grow and Prune Compact, Fast, and Accurate LSTMs
• Learning Rate Optimization for Federated Learning Exploiting Over-the-air Computation
• The Convergence of Sparsified Gradient Methods
• Taming Momentum in a Distributed Asynchronous Environment
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• JSDoop and TensorFlow.js: Volunteer Distributed Web Browser-Based Neural Network Training
• rTop-k: A Statistical Estimation Approach to Distributed SGD
• Local Stochastic Gradient Descent Ascent: Convergence Analysis and Communication Efficiency
• FedDCT: Federated Learning of Large Convolutional Neural Networks on Resource Constrained Devices using Divide and Collaborative Training
• Gradient Descent with Compressed Iterates
• SparCML: High-Performance Sparse Communication for Machine Learning
• DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling
• KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
• FedSkel: Efficient Federated Learning on Heterogeneous Systems with Skeleton Gradients Update
• Local SGD: Unified Theory and New Efficient Methods
• Faster Non-Convex Federated Learning via Global and Local Momentum
• On Communication Compression for Distributed Optimization on Heterogeneous Data
• Gradient Scheduling with Global Momentum for Non-IID Data Distributed Asynchronous Training
• One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysis
• Sparse Weight Activation Training
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• Accelerating DNN Training in Wireless Federated Edge Learning Systems
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• FedSAUC: A Similarity-Aware Update Control for Communication-Efficient Federated Learning in Edge Computing
• Adaptive Gradient Sparsification for Efficient Federated Learning: An Online Learning Approach
• TEE-based decentralized recommender systems: The raw data sharing redemption
• Wireless for Machine Learning
• FedSel: Federated SGD under Local Differential Privacy with Top-k Dimension Selection
• EventGraD: Event-Triggered Communication in Parallel Machine Learning
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Parameter-Parallel Distributed Variational Quantum Algorithm
• Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
• Elastic Consistency: A General Consistency Model for Distributed Stochastic Gradient Descent
• Fairness-aware Agnostic Federated Learning
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• OCTOPUS: Overcoming Performance andPrivatization Bottlenecks in Distributed Learning
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• ESMFL: Efficient and Secure Models for Federated Learning
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• Accelerating Distributed ML Training via Selective Synchronization
• Personalized Federated Learning by Structured and Unstructured Pruning under Data Heterogeneity
• Get More for Less in Decentralized Learning Systems
• Noiseless Privacy-Preserving Decentralized Learning
• Communication-Efficient Federated Learning with Compensated Overlap-FedAvg
• Federated Learning in Adversarial Settings
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• Distributed Training and Optimization Of Neural Networks
• Low-Rank Training of Deep Neural Networks for Emerging Memory Technology
• A Perspective on Time towards Wireless 6G
• Scheduling for Cellular Federated Edge Learning with Importance and Channel Awareness
• Trends and Advancements in Deep Neural Network Communication
• CoReS: Compatible Representations via Stationarity
• Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
• MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
• Dynamic Sparse Graph for Efficient Deep Learning
• A simplified convergence theory for Byzantine resilient stochastic gradient descent
• Distributed Machine Learning through Heterogeneous Edge Systems
• A Compressive Sensing Approach for Federated Learning over Massive MIMO Communication Systems
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Neural Machine Translation with 4-Bit Precision and Beyond
• Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
• State-of-the-art Techniques in Deep Edge Intelligence
• Compressing gradients by exploiting temporal correlation in momentum-SGD
• Collaborative Deep Learning Across Multiple Data Centers
• QLSD: Quantised Langevin stochastic dynamics for Bayesian federated learning
• SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks
• COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression
• Hydra: A Peer to Peer Distributed Training & Data Collection Framework
• Step-Ahead Error Feedback for Distributed Training with Compressed Gradient
• Accelerated Sparsified SGD with Error Feedback
• Theoretically Better and Numerically Faster Distributed Optimization with Smoothness-Aware Quantization Techniques
• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
• Adaptive Periodic Averaging: A Practical Approach to Reducing Communication in Distributed Learning
• Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• Making Asynchronous Stochastic Gradient Descent Work for Transformers
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Accelerating Training using Tensor Decomposition
• Pushing the boundaries of parallel Deep Learning -- A practical approach
• An Expectation-Maximization Perspective on Federated Learning
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• Federated Dynamic Neural Network for Deep MIMO Detection
• Slashing Communication Traffic in Federated Learning by Transmitting Clustered Model Updates
• Revolutionizing Wireless Networks with Federated Learning: A Comprehensive Review
• Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
• Batch Normalization Sampling
• MDLdroid: a ChainSGD-reduce Approach to Mobile Deep Learning for Personal Mobile Sensing
• MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning
• Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
• Quantizing data for distributed learning
• MyMigrationBot: A Cloud-based Facebook Social Chatbot for Migrant Populations
• Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters
• A study on speech enhancement using exponent-only floating point quantized neural network (EOFP-QNN)
• HPSGD: Hierarchical Parallel SGD With Stale Gradients Featuring
• QoS-Constrained Federated Learning Empowered by Intelligent Reflecting Surface
• Secure Distributed Training at Scale
• Associative Convolutional Layers
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data