Model Degradation Hinders Deep Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have achieved great success in various graph mining tasks.However, drastic performance degradation is always observed when a GNN is stacked with many layers. As a result, most GNNs only have shallow architectures, which limits their expressive power and exploitation of deep neighborhoods.Most recent studies attribute the performance degradation of deep GNNs to the \textit{over-smoothing} issue. In this paper, we disentangle the conventional graph convolution operation into two independent operations: \textit{Propagation} (\textbf{P}) and \textit{Transformation} (\textbf{T}).Following this, the depth of a GNN can be split into the propagation depth ($D_p$) and the transformation depth ($D_t$). Through extensive experiments, we find that the major cause for the performance degradation of deep GNNs is the \textit{model degradation} issue caused by large $D_t$ rather than the \textit{over-smoothing} issue mainly caused by large $D_p$. Further, we present \textit{Adaptive Initial Residual} (AIR), a plug-and-play module compatible with all kinds of GNN architectures, to alleviate the \textit{model degradation} issue and the \textit{over-smoothing} issue simultaneously. Experimental results on six real-world datasets demonstrate that GNNs equipped with AIR outperform most GNNs with shallow architectures owing to the benefits of both large $D_p$ and $D_t$, while the time costs associated with AIR can be ignored.

11 pages, 10 figures

References in corpus (10)

• Discrete Signal Processing on Graphs
• Simplifying Graph Convolutional Networks
• Towards Deeper Graph Neural Networks
• Simple and Deep Graph Convolutional Networks
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• Graph Attention Multi-Layer Perceptron
• A Note on Over-Smoothing for Graph Neural Networks
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• ROD: Reception-aware Online Distillation for Sparse Graphs
• On Provable Benefits of Depth in Training Graph Convolutional Networks

Cited by in corpus (8)

• Learning Strong Graph Neural Networks with Weak Information
• Neighborhood Homophily-based Graph Convolutional Network
• Beyond Over-smoothing: Uncovering the Trainability Challenges in Deep Graph Neural Networks
• Graph-Skeleton: ~1% Nodes are Sufficient to Represent Billion-Scale Graph
• Noise-Resilient Unsupervised Graph Representation Learning via Multi-Hop Feature Quality Estimation
• Self-supervised Representation Learning on Electronic Health Records with Graph Kernel Infomax
• Clarify Confused Nodes via Separated Learning
• Mixture of Experts Meets Decoupled Message Passing: Towards General and Adaptive Node Classification