Natural Compression for Distributed Deep Learning

Abstract

Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication of model updates among machines becomes a significant performance bottleneck and various lossy update compression techniques have been proposed to alleviate this problem. In this work, we introduce a new, simple yet theoretically and practically effective compression technique: natural compression (NC). Our technique is applied individually to all entries of the to-be-compressed update vector and works by randomized rounding to the nearest (negative or positive) power of two, which can be computed in a "natural" way by ignoring the mantissa. We show that compared to no compression, NC increases the second moment of the compressed vector by not more than the tiny factor $\frac{9}{8}$, which means that the effect of NC on the convergence speed of popular training algorithms, such as distributed SGD, is negligible. However, the communications savings enabled by NC are substantial, leading to $3$-$4\times$ improvement in overall theoretical running time. For applications requiring more aggressive compression, we generalize NC to natural dithering, which we prove is exponentially better than the common random dithering technique. Our compression operators can be used on their own or in combination with existing operators for a more aggressive combined effect and offer new state-of-the-art both in theory and practice.

Proceedings of 3${}^{\text{rd}}$ Annual Conference on Mathematical and Scientific Machine Learning (MSML 2022)

References in corpus (21)

• Federated Learning: Strategies for Improving Communication Efficiency
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Communication Efficient Distributed Optimization using an Approximate Newton-type Method
• Local SGD Converges Fast and Communicates Little
• Distributed Coordinate Descent Method for Learning with Big Data
• Distributed Mean Estimation with Limited Communication
• Scaling Distributed Machine Learning with In-Network Aggregation
• AIDE: Fast and Communication Efficient Distributed Optimization
• Not All Samples Are Created Equal: Deep Learning with Importance Sampling
• Distributed Learning with Compressed Gradient Differences
• Stochastic Distributed Learning with Gradient Quantization and Variance Reduction
• 3LC: Lightweight and Effective Traffic Compression for Distributed Machine Learning
• Distributed learning with compressed gradients
• Nonconvex Variance Reduced Optimization with Arbitrary Sampling
• 99% of Distributed Optimization is a Waste of Time: The Issue and How to Fix it

Cited by in corpus (31)

• UVeQFed: Universal Vector Quantization for Federated Learning
• A Field Guide to Federated Optimization
• Optimal Client Sampling for Federated Learning
• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey
• Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization
• FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropout
• On Biased Compression for Distributed Learning
• EF21: A New, Simpler, Theoretically Better, and Practically Faster Error Feedback
• Optimal Gradient Compression for Distributed and Federated Learning
• Adaptive Gradient Quantization for Data-Parallel SGD
• Recent theoretical advances in decentralized distributed convex optimization
• A Unified Analysis of Stochastic Gradient Methods for Nonconvex Federated Optimization
• On the Utility of Gradient Compression in Distributed Training Systems
• A Better Alternative to Error Feedback for Communication-Efficient Distributed Learning
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• MARINA: Faster Non-Convex Distributed Learning with Compression
• Gradient Descent with Compressed Iterates
• Distributed Second Order Methods with Fast Rates and Compressed Communication
• Large-Scale Deep Learning Optimizations: A Comprehensive Survey
• Uncertainty Principle for Communication Compression in Distributed and Federated Learning and the Search for an Optimal Compressor
• Distributed Fixed Point Methods with Compressed Iterates
• CANITA: Faster Rates for Distributed Convex Optimization with Communication Compression
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• Federated Learning is Better with Non-Homomorphic Encryption
• Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
• Permutation Compressors for Provably Faster Distributed Nonconvex Optimization
• Smoothness Matrices Beat Smoothness Constants: Better Communication Compression Techniques for Distributed Optimization
• Error Compensated Loopless SVRG, Quartz, and SDCA for Distributed Optimization
• Distributed Sparse SGD with Majority Voting
• FedNS: Improving Federated Learning for collaborative image classification on mobile clients
• Compressed Communication for Distributed Training: Adaptive Methods and System