Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs

Abstract

Distributed Deep Learning (DDL) has rapidly grown its popularity since it helps boost the training performance on high-performance GPU clusters. Efficient job scheduling is indispensable to maximize the overall performance of the cluster when training multiple jobs simultaneously. However, existing schedulers do not consider the communication contention of multiple communication tasks from different distributed training jobs, which could deteriorate the system performance and prolong the job completion time. In this paper, we first establish a new DDL job scheduling framework which organizes DDL jobs as Directed Acyclic Graphs (DAGs) and considers communication contention between nodes. We then propose an efficient algorithm, LWF-$\kappa$, to balance the GPU utilization and consolidate the allocated GPUs for each job. When scheduling those communication tasks, we observe that neither avoiding all the contention nor blindly accepting them is optimal to minimize the job completion time. We thus propose a provable algorithm, AdaDUAL, to efficiently schedule those communication tasks. Based on AdaDUAL, we finally propose Ada-SRSF for the DDL job scheduling problem. Simulations on a 64-GPU cluster connected with 10 Gbps Ethernet show that LWF-$\kappa$ achieves up to $1.59\times$ improvement over the classical first-fit algorithms. More importantly, Ada-SRSF reduces the average job completion time by $20.1\%$ and $36.7\%$, as compared to the SRSF(1) scheme (avoiding all the contention) and the SRSF(2) scheme (blindly accepting all of two-way communication contention) respectively.

References in corpus (8)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Horovod: fast and easy distributed deep learning in TensorFlow
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Scaling Deep Learning on GPU and Knights Landing clusters
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• Online Job Scheduling in Distributed Machine Learning Clusters
• A DAG Model of Synchronous Stochastic Gradient Descent in Distributed Deep Learning
• Low-latency job scheduling with preemption for the development of deep learning

Cited by in corpus (1)

• Communication-Efficient Distributed Deep Learning: A Comprehensive Survey