Analyzing the Performance of Graph Neural Networks with Pipe Parallelism

Abstract

Many interesting datasets ubiquitous in machine learning and deep learning can be described via graphs. As the scale and complexity of graph-structured datasets increase, such as in expansive social networks, protein folding, chemical interaction networks, and material phase transitions, improving the efficiency of the machine learning techniques applied to these is crucial. In this study, we focus on Graph Neural Networks (GNN) that have found great success in tasks such as node or edge classification and link prediction. However, standard GNN models have scaling limits due to necessary recursive calculations performed through dense graph relationships that lead to memory and runtime bottlenecks. While new approaches for processing larger networks are needed to advance graph techniques, and several have been proposed, we study how GNNs could be parallelized using existing tools and frameworks that are known to be successful in the deep learning community. In particular, we investigate applying pipeline parallelism to GNN models with GPipe, introduced by Google in 2018.

arXiv commentProceedings of the conference MLSys'21 Workshop on Graph Neural Networks and Systems (GNNSys'21), San Jose, CA, USA, 2021

References in corpus (19)

• Semi-Supervised Classification with Graph Convolutional Networks
• Cooperative Game Theory Approaches for Network Partitioning
• Inductive Representation Learning on Large Graphs
• Spectral Networks and Locally Connected Networks on Graphs
• Graph Neural Networks: A Review of Methods and Applications
• Deep Convolutional Networks on Graph-Structured Data
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Fast Graph Representation Learning with PyTorch Geometric
• Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Gated Graph Sequence Neural Networks
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• SIGN: Scalable Inception Graph Neural Networks
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• Automating Botnet Detection with Graph Neural Networks
• Architectural Implications of Graph Neural Networks
• torchgpipe: On-the-fly Pipeline Parallelism for Training Giant Models
• A Survey on Graph Neural Networks for Knowledge Graph Completion
• Neural Graph Embedding Methods for Natural Language Processing