Flexible Communication for Optimal Distributed Learning over Unpredictable Networks

Abstract

Gradient compression alleviates expensive communication in distributed deep learning by sending fewer values and its corresponding indices, typically via Allgather (AG). Training with high compression ratio (CR) achieves high accuracy like DenseSGD, but has lower parallel scaling due to high communication cost (i.e., parallel efficiency). Using lower CRs improves parallel efficiency by lowering synchronization cost, but degrades model accuracy as well (statistical efficiency). Further, speedup attained with different models and CRs also varies with network latency, effective bandwidth and collective op used for aggregation. In many cases, collectives like Allreduce (AR) have lower cost than AG to exchange the same amount of data. In this paper, we propose an AR-compatible Topk compressor that is bandwidth-optimal and thus performs better than AG in certain network configurations. We develop a flexible communication strategy that switches between AG and AR based on which collective is optimal in the current settings, and model the pareto-relationship between parallel and statistical efficiency as a multi-objective optimization (MOO) problem to dynamically adjust CR and accelerate training while still converging to high accuracy.

arXiv comment2023 IEEE International Conference on Big Data (BigData)

References in corpus (17)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• pymoo: Multi-objective Optimization in Python
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
• An Empirical Model of Large-Batch Training
• Critical Learning Periods in Deep Neural Networks
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• Towards Scalable Distributed Training of Deep Learning on Public Cloud Clusters
• AdaScale SGD: A User-Friendly Algorithm for Distributed Training
• On the Utility of Gradient Compression in Distributed Training Systems
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• ScaDLES: Scalable Deep Learning over Streaming data at the Edge
• Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training
• GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Training
• Accelerating Distributed ML Training via Selective Synchronization