GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation

Abstract

The existing graph neural networks (GNNs) based on the spectral graph convolutional operator have been criticized for its performance degradation, which is especially common for the models with deep architectures. In this paper, we further identify the suspended animation problem with the existing GNNs. Such a problem happens when the model depth reaches the suspended animation limit, and the model will not respond to the training data any more and become not learnable. Analysis about the causes of the suspended animation problem with existing GNNs will be provided in this paper, whereas several other peripheral factors that will impact the problem will be reported as well. To resolve the problem, we introduce the GResNet (Graph Residual Network) framework in this paper, which creates extensively connected highways to involve nodes' raw features or intermediate representations throughout the graph for all the model layers. Different from the other learning settings, the extensive connections in the graph data will render the existing simple residual learning methods fail to work. We prove the effectiveness of the introduced new graph residual terms from the norm preservation perspective, which will help avoid dramatic changes to the node's representations between sequential layers. Detailed studies about the GResNet framework for many existing GNNs, including GCN, GAT and LoopyNet, will be reported in the paper with extensive empirical experiments on real-world benchmark datasets.

18 pages

References in corpus (9)

• Semi-Supervised Classification with Graph Convolutional Networks
• Deep Residual Learning for Image Recognition
• Graph Neural Networks: A Review of Methods and Applications
• Dilated Residual Networks
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Deep Pyramidal Residual Networks
• Convolutional Set Matching for Graph Similarity
• Representing smooth functions as compositions of near-identity functions with implications for deep network optimization
• Deep Loopy Neural Network Model for Graph Structured Data Representation Learning

Cited by in corpus (16)

• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• Differentiable Rendering: A Survey
• Graph Neural Ordinary Differential Equations
• Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning
• Dissecting the Diffusion Process in Linear Graph Convolutional Networks
• Ripple Walk Training: A Subgraph-based training framework for Large and Deep Graph Neural Network
• Learning Attributed Graph Representations with Communicative Message Passing Transformer
• Deoscillated Graph Collaborative Filtering
• Segmented Graph-Bert for Graph Instance Modeling
• Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks
• Graph Neural Distance Metric Learning with Graph-Bert
• Get Rid of Suspended Animation Problem: Deep Diffusive Neural Network on Graph Semi-Supervised Classification
• G5: A Universal GRAPH-BERT for Graph-to-Graph Transfer and Apocalypse Learning
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities
• Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning
• Multimodal Graph-based Transformer Framework for Biomedical Relation Extraction