Gradient Sparsification for Communication-Efficient Distributed Optimization

Abstract

Modern large scale machine learning applications require stochastic optimization algorithms to be implemented on distributed computational architectures. A key bottleneck is the communication overhead for exchanging information such as stochastic gradients among different workers. In this paper, to reduce the communication cost we propose a convex optimization formulation to minimize the coding length of stochastic gradients. To solve the optimal sparsification efficiently, several simple and fast algorithms are proposed for approximate solution, with theoretical guaranteed for sparseness. Experiments on ${\ell }_2$ regularized logistic regression, support vector machines, and convolutional neural networks validate our sparsification approaches.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
• AIDE: Fast and Communication Efficient Distributed Optimization
• On Variance Reduction in Stochastic Gradient Descent and its Asynchronous Variants
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• Improved Optimization of Finite Sums with Minibatch Stochastic Variance Reduced Proximal Iterations

Cited by in corpus (23)

• Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• No One Left Behind: Inclusive Federated Learning over Heterogeneous Devices
• Communication optimization strategies for distributed deep neural network training: A survey
• DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
• Towards Communication-efficient and Attack-Resistant Federated Edge Learning for Industrial Internet of Things
• MATCHA: Speeding Up Decentralized SGD via Matching Decomposition Sampling
• Federated Few-Shot Learning for Mobile NLP
• Convergence of Distributed Stochastic Variance Reduced Methods without Sampling Extra Data
• BROADCAST: Reducing Both Stochastic and Compression Noise to Robustify Communication-Efficient Federated Learning
• Secure Aggregation with Heterogeneous Quantization in Federated Learning
• The Convergence of Sparsified Gradient Methods
• Gradient Descent with Compressed Iterates
• Multi-node Bert-pretraining: Cost-efficient Approach
• On Consensus-Optimality Trade-offs in Collaborative Deep Learning
• Distributed Inexact Successive Convex Approximation ADMM: Analysis-Part I
• Compressing gradients by exploiting temporal correlation in momentum-SGD
• Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression
• Federated Learning is Better with Non-Homomorphic Encryption
• Compressed Distributed Gradient Descent: Communication-Efficient Consensus over Networks
• Sparsification as a Remedy for Staleness in Distributed Asynchronous SGD
• Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient
• Speeding-Up Back-Propagation in DNN: Approximate Outer Product with Memory