PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm

Abstract

Graph neural networks (GNNs) have achieved state-of-the-art performance in various graph-based tasks. However, as mainstream GNNs are designed based on the neural message passing mechanism, they do not scale well to data size and message passing steps. Although there has been an emerging interest in the design of scalable GNNs, current researches focus on specific GNN design, rather than the general design space, limiting the discovery of potential scalable GNN models. This paper proposes PasCa, a new paradigm and system that offers a principled approach to systemically construct and explore the design space for scalable GNNs, rather than studying individual designs. Through deconstructing the message passing mechanism, PasCa presents a novel Scalable Graph Neural Architecture Paradigm (SGAP), together with a general architecture design space consisting of 150k different designs. Following the paradigm, we implement an auto-search engine that can automatically search well-performing and scalable GNN architectures to balance the trade-off between multiple criteria (e.g., accuracy and efficiency) via multi-objective optimization. Empirical studies on ten benchmark datasets demonstrate that the representative instances (i.e., PasCa-V1, V2, and V3) discovered by our system achieve consistent performance among competitive baselines. Concretely, PasCa-V3 outperforms the state-of-the-art GNN method JK-Net by 0.4\% in terms of predictive accuracy on our large industry dataset while achieving up to $28.3\times$ training speedups.

arXiv comment13 pages, 8 figures. arXiv admin note: text overlap with arXiv:2104.09880

References in corpus (28)

• A Comprehensive Survey on Graph Neural Networks
• Geometric deep learning: going beyond Euclidean data
• Graph Convolutional Neural Networks for Web-Scale Recommender Systems
• AutoML: A Survey of the State-of-the-Art
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Simplifying Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• Pitfalls of Graph Neural Network Evaluation
• DeepInf: Social Influence Prediction with Deep Learning
• Predict then Propagate: Graph Neural Networks meet Personalized PageRank
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• Geometric Matrix Completion with Recurrent Multi-Graph Neural Networks
• Spam Review Detection with Graph Convolutional Networks
• SIGN: Scalable Inception Graph Neural Networks
• Design Space for Graph Neural Networks
• Graph Attention Multi-Layer Perceptron
• Graph Neural Networks in Recommender Systems: A Survey
• Adaptive Propagation Graph Convolutional Network
• OpenBox: A Generalized Black-box Optimization Service
• Scalable Graph Neural Networks via Bidirectional Propagation
• Graph Partition Neural Networks for Semi-Supervised Classification
• Hashing-Accelerated Graph Neural Networks for Link Prediction
• Efficient End-to-End AutoML via Scalable Search Space Decomposition
• ROD: Reception-aware Online Distillation for Sparse Graphs
• VolcanoML: Speeding up End-to-End AutoML via Scalable Search Space Decomposition
• Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads
• Learning to Identify High Betweenness Centrality Nodes from Scratch: A Novel Graph Neural Network Approach
• Evaluating Deep Graph Neural Networks

Cited by in corpus (5)

• Graph Attention Multi-Layer Perceptron
• BlindFL: Vertical Federated Machine Learning without Peeking into Your Data
• Model Degradation Hinders Deep Graph Neural Networks
• Robust Mid-Pass Filtering Graph Convolutional Networks
• Exploring Robust Architectures for Deep Artificial Neural Networks