Model Degradation Hinders Deep Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have achieved great success in various graph mining tasks.However, drastic performance degradation is always observed when a GNN is stacked with many layers. As a result, most GNNs only have shallow architectures, which limits their expressive power and exploitation of deep neighborhoods.Most recent studies attribute the performance degradation of deep GNNs to the \textit{over-smoothing} issue. In this paper, we disentangle the conventional graph convolution operation into two independent operations: \textit{Propagation} (\textbf{P}) and \textit{Transformation} (\textbf{T}).Following this, the depth of a GNN can be split into the propagation depth ($D_p$) and the transformation depth ($D_t$). Through extensive experiments, we find that the major cause for the performance degradation of deep GNNs is the \textit{model degradation} issue caused by large $D_t$ rather than the \textit{over-smoothing} issue mainly caused by large $D_p$. Further, we present \textit{Adaptive Initial Residual} (AIR), a plug-and-play module compatible with all kinds of GNN architectures, to alleviate the \textit{model degradation} issue and the \textit{over-smoothing} issue simultaneously. Experimental results on six real-world datasets demonstrate that GNNs equipped with AIR outperform most GNNs with shallow architectures owing to the benefits of both large $D_p$ and $D_t$, while the time costs associated with AIR can be ignored.

arXiv comment11 pages, 10 figures

References in corpus (24)

• Spectral Networks and Locally Connected Networks on Graphs
• Discrete Signal Processing on Graphs
• Simplifying Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• DeepInf: Social Influence Prediction with Deep Learning
• Towards Deeper Graph Neural Networks
• Predict then Propagate: Graph Neural Networks meet Personalized PageRank
• LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation
• Simple and Deep Graph Convolutional Networks
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• PairNorm: Tackling Oversmoothing in GNNs
• SIGN: Scalable Inception Graph Neural Networks
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Graph Attention Multi-Layer Perceptron
• Graph Neural Networks in Recommender Systems: A Survey
• Adaptive Propagation Graph Convolutional Network
• A Note on Over-Smoothing for Graph Neural Networks
• Factorizable Graph Convolutional Networks
• Scalable Graph Neural Networks via Bidirectional Propagation
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• ROD: Reception-aware Online Distillation for Sparse Graphs
• On Provable Benefits of Depth in Training Graph Convolutional Networks

Cited by in corpus (8)

• Learning Strong Graph Neural Networks with Weak Information
• Neighborhood Homophily-based Graph Convolutional Network
• Beyond Over-smoothing: Uncovering the Trainability Challenges in Deep Graph Neural Networks
• Graph-Skeleton: ~1% Nodes are Sufficient to Represent Billion-Scale Graph
• Noise-Resilient Unsupervised Graph Representation Learning via Multi-Hop Feature Quality Estimation
• Self-supervised Representation Learning on Electronic Health Records with Graph Kernel Infomax
• Clarify Confused Nodes via Separated Learning
• Mixture of Experts Meets Decoupled Message Passing: Towards General and Adaptive Node Classification