Graph Attention Multi-Layer Perceptron

Abstract

Graph neural networks (GNNs) have achieved great success in many graph-based applications. However, the enormous size and high sparsity level of graphs hinder their applications under industrial scenarios. Although some scalable GNNs are proposed for large-scale graphs, they adopt a fixed $K$-hop neighborhood for each node, thus facing the over-smoothing issue when adopting large propagation depths for nodes within sparse regions. To tackle the above issue, we propose a new GNN architecture -- Graph Attention Multi-Layer Perceptron (GAMLP), which can capture the underlying correlations between different scales of graph knowledge. We have deployed GAMLP in Tencent with the Angel platform, and we further evaluate GAMLP on both real-world datasets and large-scale industrial datasets. Extensive experiments on these 14 graph datasets demonstrate that GAMLP achieves state-of-the-art performance while enjoying high scalability and efficiency. Specifically, it outperforms GAT by 1.3\% regarding predictive accuracy on our large-scale Tencent Video dataset while achieving up to $50\times$ training speedup. Besides, it ranks top-1 on both the leaderboards of the largest homogeneous and heterogeneous graph (i.e., ogbn-papers100M and ogbn-mag) of Open Graph Benchmark.

11 pages, 7 figures. arXiv admin note: text overlap with arXiv:2108.10097

References in corpus (20)

• Semi-Supervised Classification with Graph Convolutional Networks
• An Overview of Multi-Task Learning in Deep Neural Networks
• Fast Graph Representation Learning with PyTorch Geometric
• Simplifying Graph Convolutional Networks
• Simple and Deep Graph Convolutional Networks
• DeeperGCN: All You Need to Train Deeper GCNs
• Adaptive Graph Encoder for Attributed Graph Embedding
• OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
• Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
• Heterogeneous Graph Representation Learning with Relation Awareness
• When Does Self-Supervision Help Graph Convolutional Networks?
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• Bag of Tricks for Node Classification with Graph Neural Networks
• Scalable Graph Neural Networks for Heterogeneous Graphs
• ROD: Reception-aware Online Distillation for Sparse Graphs
• Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training
• Evaluating Deep Graph Neural Networks
• Hybrid Micro/Macro Level Convolution for Heterogeneous Graph Learning
• R-GSN: The Relation-based Graph Similar Network for Heterogeneous Graph
• GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing

Cited by in corpus (11)

• Attention-based graph neural networks: a survey
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• Learning Strong Graph Neural Networks with Weak Information
• Model Degradation Hinders Deep Graph Neural Networks
• Efficient Heterogeneous Graph Learning via Random Projection
• Topology-aware Embedding Memory for Continual Learning on Expanding Networks
• Seq-HGNN: Learning Sequential Node Representation on Heterogeneous Graph
• Noise-Resilient Unsupervised Graph Representation Learning via Multi-Hop Feature Quality Estimation
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
• Boosting Multitask Learning on Graphs through Higher-Order Task Affinities
• Visualization and Analysis of the Loss Landscape in Graph Neural Networks