Evaluating Deep Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have already been widely applied in various graph mining tasks. However, they suffer from the shallow architecture issue, which is the key impediment that hinders the model performance improvement. Although several relevant approaches have been proposed, none of the existing studies provides an in-depth understanding of the root causes of performance degradation in deep GNNs. In this paper, we conduct the first systematic experimental evaluation to present the fundamental limitations of shallow architectures. Based on the experimental results, we answer the following two essential questions: (1) what actually leads to the compromised performance of deep GNNs; (2) when we need and how to build deep GNNs. The answers to the above questions provide empirical insights and guidelines for researchers to design deep and well-performed GNNs. To show the effectiveness of our proposed guidelines, we present Deep Graph Multi-Layer Perceptron (DGMLP), a powerful approach (a paradigm in its own right) that helps guide deep GNN designs. Experimental results demonstrate three advantages of DGMLP: 1) high accuracy -- it achieves state-of-the-art node classification performance on various datasets; 2) high flexibility -- it can flexibly choose different propagation and transformation depths according to graph size and sparsity; 3) high scalability and efficiency -- it supports fast training on large-scale graphs. Our code is available in https://github.com/zwt233/DGMLP.

arXiv comment14 pages, 14 figures, 7 tables

References in corpus (24)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Graph Neural Networks: A Review of Methods and Applications
• Simplifying Graph Convolutional Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Towards Deeper Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Few-Shot Learning with Graph Neural Networks
• Simple and Deep Graph Convolutional Networks
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• SIGN: Scalable Inception Graph Neural Networks
• Adaptive Propagation Graph Convolutional Network
• A Note on Over-Smoothing for Graph Neural Networks
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization
• Graph Neural Networks for Natural Language Processing: A Survey
• Revisiting Over-smoothing in Deep GCNs
• Retrosynthesis Prediction with Conditional Graph Logic Network
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks
• DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs
• ROD: Reception-aware Online Distillation for Sparse Graphs
• Deep Graph Neural Networks with Shallow Subgraph Samplers
• GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passing

Cited by in corpus (3)

• Graph Attention Multi-Layer Perceptron
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• Node Dependent Local Smoothing for Scalable Graph Learning