Towards Deeper Graph Neural Networks

Abstract

Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, co-authorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods.

11 pages, KDD2020

References in corpus (10)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering
• Fast Graph Representation Learning with PyTorch Geometric
• Simplifying Graph Convolutional Networks
• Pitfalls of Graph Neural Network Evaluation
• Large-Scale Learnable Graph Convolutional Networks
• Link Prediction Based on Graph Neural Networks
• Geom-GCN: Geometric Graph Convolutional Networks

Cited by in corpus (75)

• Graph Contrastive Learning with Augmentations
• Bootstrap Latent Representations for Multi-modal Recommendation
• Graph Neural Networks: Taxonomy, Advances and Trends
• DGRec: Graph Neural Network for Recommendation with Diversified Embedding Generation
• Attention-based graph neural networks: a survey
• Improving Fairness in Graph Neural Networks via Mitigating Sensitive Attribute Leakage
• Multi-Behavior Sequential Recommendation with Temporal Graph Transformer
• Condensing Graphs via One-Step Gradient Matching
• Investigating Accuracy-Novelty Performance for Graph-based Collaborative Filtering
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• GraphAIR: Graph Representation Learning with Neighborhood Aggregation and Interaction
• Graph-less Collaborative Filtering
• AdaGNN: Graph Neural Networks with Adaptive Frequency Response Filter
• Motif-based Graph Self-Supervised Learning for Molecular Property Prediction
• Model Degradation Hinders Deep Graph Neural Networks
• Company-as-Tribe: Company Financial Risk Assessment on Tribe-Style Graph with Hierarchical Graph Neural Networks
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective
• AdaGCN: Adaboosting Graph Convolutional Networks into Deep Models
• BernNet: Learning Arbitrary Graph Spectral Filters via Bernstein Approximation
• Domain-adaptive Message Passing Graph Neural Network
• Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective
• Interpreting and Unifying Graph Neural Networks with An Optimization Framework
• Learn molecular representations from large-scale unlabeled molecules for drug discovery
• HighAir: A Hierarchical Graph Neural Network-Based Air Quality Forecasting Method
• Meta-Weight Graph Neural Network: Push the Limits Beyond Global Homophily
• MM-GNN: Mix-Moment Graph Neural Network towards Modeling Neighborhood Feature Distribution
• DGNN: Decoupled Graph Neural Networks with Structural Consistency between Attribute and Graph Embedding Representations
• Beyond Graph Convolutional Network: An Interpretable Regularizer-centered Optimization Framework
• Evaluating Deep Graph Neural Networks
• Residual Network and Embedding Usage: New Tricks of Node Classification with Graph Convolutional Networks
• Deep Outdated Fact Detection in Knowledge Graphs
• Optimal Solving of Constrained Path-Planning Problems with Graph Convolutional Networks and Optimized Tree Search
• Predicting the Silent Majority on Graphs: Knowledge Transferable Graph Neural Network
• Node Similarity Preserving Graph Convolutional Networks
• NODE-SELECT: A Graph Neural Network Based On A Selective Propagation Technique
• Sampling methods for efficient training of graph convolutional networks: A survey
• Graph Neural Networks Inspired by Classical Iterative Algorithms
• Deep Graph Neural Networks with Shallow Subgraph Samplers
• Node Dependent Local Smoothing for Scalable Graph Learning
• Low-bit Quantization for Deep Graph Neural Networks with Smoothness-aware Message Propagation
• Multi-relational Graph Diffusion Neural Network with Parallel Retention for Stock Trends Classification
• DGDNN: Decoupled Graph Diffusion Neural Network for Stock Movement Prediction
• Multi-View Subgraph Neural Networks: Self-Supervised Learning with Scarce Labeled Data
• FedGL: Federated Graph Learning Framework with Global Self-Supervision
• Improving the Expressiveness of $K$-hop Message-Passing GNNs by Injecting Contextualized Substructure Information
• On the Equivalence of Decoupled Graph Convolution Network and Label Propagation
• Knowledge-aware Contrastive Molecular Graph Learning
• Robust Node Classification on Graphs: Jointly from Bayesian Label Transition and Topology-based Label Propagation
• Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• FairSample: Training Fair and Accurate Graph Convolutional Neural Networks Efficiently
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• A Survey on Learning from Graphs with Heterophily: Recent Advances and Future Directions
• Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs
• Instance-wise Graph-based Framework for Multivariate Time Series Forecasting
• Fast Quantum Property Prediction via Deeper 2D and 3D Graph Networks
• Adversarial Graph Disentanglement
• Tree Decomposed Graph Neural Network
• GraTO: Graph Neural Network Framework Tackling Over-smoothing with Neural Architecture Search
• A Comprehensive Benchmark on Spectral GNNs: The Impact on Efficiency, Memory, and Effectiveness
• Topological Regularization for Graph Neural Networks Augmentation
• HyperGALE: ASD Classification via Hypergraph Gated Attention with Learnable Hyperedges
• Graph Belief Propagation Networks
• LEReg: Empower Graph Neural Networks with Local Energy Regularization
• AKE-GNN: Effective Graph Learning with Adaptive Knowledge Exchange
• Parallel Multi-Graph Convolution Network For Metro Passenger Volume Prediction
• Virtual Nodes based Heterogeneous Graph Convolutional Neural Network for Efficient Long-Range Information Aggregation
• Learning Parametrised Graph Shift Operators
• Cross-Modality Protein Embedding for Compound-Protein Affinity and Contact Prediction
• Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks
• Scaling Up Graph Neural Networks Via Graph Coarsening
• Alleviating Over-Smoothing via Aggregation over Compact Manifolds
• Graph Feature Gating Networks