TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

Abstract

High network communication cost for synchronizing gradients and parameters is the well-known bottleneck of distributed training. In this work, we propose TernGrad that uses ternary gradients to accelerate distributed deep learning in data parallelism. Our approach requires only three numerical levels {-1,0,1}, which can aggressively reduce the communication time. We mathematically prove the convergence of TernGrad under the assumption of a bound on gradients. Guided by the bound, we propose layer-wise ternarizing and gradient clipping to improve its convergence. Our experiments show that applying TernGrad on AlexNet does not incur any accuracy loss and can even improve accuracy. The accuracy loss of GoogLeNet induced by TernGrad is less than 2% on average. Finally, a performance model is proposed to study the scalability of TernGrad. Experiments show significant speed gains for various deep neural networks. Our source code is available.

NIPS 2017 Oral

Cited by in corpus (163)

• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Training and Inference with Integers in Deep Neural Networks
• UVeQFed: Universal Vector Quantization for Federated Learning
• Local SGD Converges Fast and Communicates Little
• A Survey on Methods and Theories of Quantized Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Database Meets Deep Learning: Challenges and Opportunities
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Deep Learning in Mobile and Wireless Networking: A Survey
• Scaling Distributed Machine Learning with In-Network Aggregation
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Distributed Learning with Compressed Gradient Differences
• Optimal Client Sampling for Federated Learning
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Fast Federated Learning by Balancing Communication Trade-Offs
• On Maintaining Linear Convergence of Distributed Learning and Optimization under Limited Communication
• The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Communication
• Communication optimization strategies for distributed deep neural network training: A survey
• Communication-Efficient Distributed Blockwise Momentum SGD with Error-Feedback
• Natural Compression for Distributed Deep Learning
• Pipe-SGD: A Decentralized Pipelined SGD Framework for Distributed Deep Net Training
• Near-Optimal Sparse Allreduce for Distributed Deep Learning
• PruneTrain: Fast Neural Network Training by Dynamic Sparse Model Reconfiguration
• Variance-based Gradient Compression for Efficient Distributed Deep Learning
• DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Priority-based Parameter Propagation for Distributed DNN Training
• Distributed Deep Reinforcement Learning: A Survey and A Multi-Player Multi-Agent Learning Toolbox
• RedSync : Reducing Synchronization Traffic for Distributed Deep Learning
• Distributed learning with compressed gradients
• Linear Convergence in Federated Learning: Tackling Client Heterogeneity and Sparse Gradients
• Optimal Gradient Compression for Distributed and Federated Learning
• A Federated Deep Learning Framework for Privacy Preservation and Communication Efficiency
• Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization
• Breaking the Communication-Privacy-Accuracy Trilemma
• Moniqua: Modulo Quantized Communication in Decentralized SGD
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• Decentralized Deep Learning with Arbitrary Communication Compression
• A Double Residual Compression Algorithm for Efficient Distributed Learning
• Adaptive Gradient Quantization for Data-Parallel SGD
• Federated Learning with Compression: Unified Analysis and Sharp Guarantees
• Recent theoretical advances in decentralized distributed convex optimization
• Decentralized Learning of Generative Adversarial Networks from Non-iid Data
• Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
• Election Coding for Distributed Learning: Protecting SignSGD against Byzantine Attacks
• A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
• Communication-Efficient Robust Federated Learning with Noisy Labels
• $\text{DeepSqueeze}$: Decentralization Meets Error-Compensated Compression
• Reducing Communication for Split Learning by Randomized Top-k Sparsification
• Federated Accelerated Stochastic Gradient Descent
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• On the Utility of Gradient Compression in Distributed Training Systems
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Communication trade-offs for synchronized distributed SGD with large step size
• A Unified Theory of SGD: Variance Reduction, Sampling, Quantization and Coordinate Descent
• BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
• Backprop with Approximate Activations for Memory-efficient Network Training
• Communication-efficient distributed SGD with Sketching
• 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
• Learning Rate Optimization for Federated Learning Exploiting Over-the-air Computation
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Ternary Compression for Communication-Efficient Federated Learning
• ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training
• The Convergence of Sparsified Gradient Methods
• Brainstorming Generative Adversarial Networks (BGANs): Towards Multi-Agent Generative Models with Distributed Private Datasets
• Parallel Restarted SPIDER -- Communication Efficient Distributed Nonconvex Optimization with Optimal Computation Complexity
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Taming Momentum in a Distributed Asynchronous Environment
• rTop-k: A Statistical Estimation Approach to Distributed SGD
• JSDoop and TensorFlow.js: Volunteer Distributed Web Browser-Based Neural Network Training
• Gradient Descent with Compressed Iterates
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• DynaComm: Accelerating Distributed CNN Training between Edges and Clouds through Dynamic Communication Scheduling
• FedSKETCH: Communication-Efficient and Private Federated Learning via Sketching
• SparCML: High-Performance Sparse Communication for Machine Learning
• Efficient Sparse Secure Aggregation for Federated Learning
• DRIVE: One-bit Distributed Mean Estimation
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• On Communication Compression for Distributed Optimization on Heterogeneous Data
• Wyner-Ziv Gradient Compression for Federated Learning
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• Accelerating CNN Training by Pruning Activation Gradients
• PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning
• Faster Neural Network Training with Approximate Tensor Operations
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
• Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
• Improved Convergence Analysis and SNR Control Strategies for Federated Learning in the Presence of Noise
• ESMFL: Efficient and Secure Models for Federated Learning
• Distributed Additive Encryption and Quantization for Privacy Preserving Federated Deep Learning
• DBS: Dynamic Batch Size For Distributed Deep Neural Network Training
• Accelerating Distributed ML Training via Selective Synchronization
• Distributed Newton Can Communicate Less and Resist Byzantine Workers
• Federated Learning in Adversarial Settings
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• On Consensus-Optimality Trade-offs in Collaborative Deep Learning
• New Bounds For Distributed Mean Estimation and Variance Reduction
• Compressed Federated Reinforcement Learning with a Generative Model
• MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
• Dynamic Sparse Graph for Efficient Deep Learning
• Faster Distributed Deep Net Training: Computation and Communication Decoupled Stochastic Gradient Descent
• Trends and Advancements in Deep Neural Network Communication
• Communication Efficient Federated Learning with Energy Awareness over Wireless Networks
• ShadowSync: Performing Synchronization in the Background for Highly Scalable Distributed Training
• Mesa: A Memory-saving Training Framework for Transformers
• Decentralized Optimization On Time-Varying Directed Graphs Under Communication Constraints
• SuperNeurons: FFT-based Gradient Sparsification in the Distributed Training of Deep Neural Networks
• Federated Learning is Better with Non-Homomorphic Encryption
• A flexible framework for communication-efficient machine learning: from HPC to IoT
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• Compressing gradients by exploiting temporal correlation in momentum-SGD
• An End-to-End Encrypted Neural Network for Gradient Updates Transmission in Federated Learning
• 1-bit LAMB: Communication Efficient Large-Scale Large-Batch Training with LAMB's Convergence Speed
• Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
• Constrained Differentially Private Federated Learning for Low-bandwidth Devices
• CGX: Adaptive System Support for Communication-Efficient Deep Learning
• Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning
• Toward Model Parallelism for Deep Neural Network based on Gradient-free ADMM Framework
• Step-Ahead Error Feedback for Distributed Training with Compressed Gradient
• Adaptive Periodic Averaging: A Practical Approach to Reducing Communication in Distributed Learning
• Adaptive Gradient Coding
• Quantized Adam with Error Feedback
• A Hybrid-Order Distributed SGD Method for Non-Convex Optimization to Balance Communication Overhead, Computational Complexity, and Convergence Rate
• Accelerated CNN Training Through Gradient Approximation
• Accelerated Sparsified SGD with Error Feedback
• Flexible Communication for Optimal Distributed Learning over Unpredictable Networks
• Compressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks
• Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
• CFedAvg: Achieving Efficient Communication and Fast Convergence in Non-IID Federated Learning
• A Distributed Training Algorithm of Generative Adversarial Networks with Quantized Gradients
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Distributed Sparse SGD with Majority Voting
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• On the Convergence of Quantized Parallel Restarted SGD for Central Server Free Distributed Training
• CSER: Communication-efficient SGD with Error Reset
• Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
• Compressed Coded Distributed Computing
• Enabling Binary Neural Network Training on the Edge
• Gradient Sparification for Asynchronous Distributed Training
• CD-SGD: Distributed Stochastic Gradient Descent with Compression and Delay Compensation
• Gap Aware Mitigation of Gradient Staleness
• Approximate Random Dropout
• Reducing Data Motion to Accelerate the Training of Deep Neural Networks
• Caramel: Accelerating Decentralized Distributed Deep Learning with Computation Scheduling
• Quantizing data for distributed learning
• MURANA: A Generic Framework for Stochastic Variance-Reduced Optimization
• Progressive Compressed Records: Taking a Byte out of Deep Learning Data
• MG-WFBP: Merging Gradients Wisely for Efficient Communication in Distributed Deep Learning
• Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
• SparseTrain: Exploiting Dataflow Sparsity for Efficient Convolutional Neural Networks Training
• Edge Artificial Intelligence for 6G: Vision, Enabling Technologies, and Applications
• Scalable Projection-Free Optimization
• Doing More by Doing Less: How Structured Partial Backpropagation Improves Deep Learning Clusters
• Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework
• Optimal Compression of Locally Differentially Private Mechanisms