Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration

Abstract

Training Graph Neural Networks (GNN) on large graphs is resource-intensive and time-consuming, mainly due to the large graph data that cannot be fit into the memory of a single machine, but have to be fetched from distributed graph storage and processed on the go. Unlike distributed deep neural network (DNN) training, the bottleneck in distributed GNN training lies largely in large graph data transmission for constructing mini-batches of training samples. Existing solutions often advocate data-computation colocation, and do not work well with limited resources where the colocation is infeasible. The potentials of strategical task placement and optimal scheduling of data transmission and task execution have not been well explored. This paper designs an efficient algorithm framework for task placement and execution scheduling of distributed GNN training, to better resource utilization, improve execution pipelining, and expediting training completion. Our framework consists of two modules: (i) an online scheduling algorithm that schedules the execution of training tasks, and the data transmission plan; and (ii) an exploratory task placement scheme that decides the placement of each training task. We conduct thorough theoretical analysis, testbed experiments and simulation studies, and observe up to 67% training speed-up with our algorithm as compared to representative baselines.

arXiv commentINFOCOM 2022

References in corpus (7)

• TensorFlow: A system for large-scale machine learning
• Neural Message Passing for Quantum Chemistry
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• RouteNet: Leveraging Graph Neural Networks for network modeling and optimization in SDN
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• Device Placement Optimization with Reinforcement Learning