Sign Bit is Enough: A Learning Synchronization Framework for Multi-hop All-reduce with Ultimate Compression

Abstract

Traditional one-bit compressed stochastic gradient descent can not be directly employed in multi-hop all-reduce, a widely adopted distributed training paradigm in network-intensive high-performance computing systems such as public clouds. According to our theoretical findings, due to the cascading compression, the training process has considerable deterioration on the convergence performance. To overcome this limitation, we implement a sign-bit compression-based learning synchronization framework, Marsit. It prevents cascading compression via an elaborate bit-wise operation for unbiased sign aggregation and its specific global compensation mechanism for mitigating compression deviation. The proposed framework retains the same theoretical convergence rate as non-compression mechanisms. Experimental results demonstrate that Marsit reduces up to 35% training time while preserving the same accuracy as training without compression.

arXiv commentAccepted by DAC 2022

References in corpus (15)

• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Horovod: fast and easy distributed deep learning in TensorFlow
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Gradient Sparsification for Communication-Efficient Distributed Optimization
• Error Feedback Fixes SignSGD and other Gradient Compression Schemes
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• signSGD with Majority Vote is Communication Efficient And Fault Tolerant
• signSGD: Compressed Optimisation for Non-Convex Problems
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data
• GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training
• Stochastic Sign Descent Methods: New Algorithms and Better Theory
• Accelerating Gossip SGD with Periodic Global Averaging
• Optimal Complexity in Decentralized Training