DeeperGCN: All You Need to Train Deeper GCNs

Abstract

Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. These challenges limit the representation power of GCNs on large-scale graphs. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs. We define differentiable generalized aggregation functions to unify different message aggregation operations (e.g. mean, max). We also propose a novel normalization layer namely MsgNorm and a pre-activation version of residual connections for GCNs. Extensive experiments on Open Graph Benchmark (OGB) show DeeperGCN significantly boosts performance over the state-of-the-art on the large scale graph learning tasks of node property prediction and graph property prediction. Please visit https://www.deepgcns.org for more information.

arXiv commentThis work is still working in process. More results will be updated in the future version. Project website: https://www.deepgcns.org

References in corpus (19)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Deep Residual Learning for Image Recognition
• Inductive Representation Learning on Large Graphs
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Convolutional Networks on Graph-Structured Data
• Fast Graph Representation Learning with PyTorch Geometric
• Representation Learning on Graphs with Jumping Knowledge Networks
• Learning Convolutional Neural Networks for Graphs
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• DeepGCNs: Making GCNs Go as Deep as CNNs
• On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning
• Stochastic Training of Graph Convolutional Networks with Variance Reduction
• DeepGCNs: Can GCNs Go as Deep as CNNs?
• Invariant and Equivariant Graph Networks
• Adaptive Graph Convolutional Neural Networks
• Geometric deep learning on graphs and manifolds using mixture model CNNs

Cited by in corpus (64)

• A Comprehensive Survey on Deep Graph Representation Learning
• Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
• DeepGCNs: Making GCNs Go as Deep as CNNs
• Do Transformers Really Perform Bad for Graph Representation?
• Graph Attention Multi-Layer Perceptron
• Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification
• Global Self-Attention as a Replacement for Graph Convolution
• Variational Graph Normalized Auto-Encoders
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• Label Information Enhanced Fraud Detection against Low Homophily in Graphs
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Real-time Seismic Intensity Prediction using Self-supervised Contrastive GNN for Earthquake Early Warning
• Nested Graph Neural Networks
• Bag of Tricks for Node Classification with Graph Neural Networks
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Graph Neural Networks with Learnable Structural and Positional Representations
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• CGNN: Traffic Classification with Graph Neural Network
• Motif-Driven Contrastive Learning of Graph Representations
• Dual-view Molecule Pre-training
• Residual Network and Embedding Usage: New Tricks of Node Classification with Graph Convolutional Networks
• Graph Neural Networks Inspired by Classical Iterative Algorithms
• Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
• Explicit Feature Interaction-aware Graph Neural Networks
• Parameterized Hypercomplex Graph Neural Networks for Graph Classification
• Deep Graph Neural Networks with Shallow Subgraph Samplers
• Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking
• Learnable Graph Matching: A Practical Paradigm for Data Association
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• How Framelets Enhance Graph Neural Networks
• LibAUC: A Deep Learning Library for X-Risk Optimization
• On Graph Neural Networks versus Graph-Augmented MLPs
• Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
• MAAS: Multi-modal Assignation for Active Speaker Detection
• Network In Graph Neural Network
• HyperSAGE: Generalizing Inductive Representation Learning on Hypergraphs
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Multi-task Graph Convolutional Neural Network for Calcification Morphology and Distribution Analysis in Mammograms
• Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs
• R-GSN: The Relation-based Graph Similar Network for Heterogeneous Graph
• Graph convolutions that can finally model local structure
• Fast Quantum Property Prediction via Deeper 2D and 3D Graph Networks
• Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks
• Graph Coordinates and Conventional Neural Networks -- An Alternative for Graph Neural Networks
• Wasserstein Embedding for Graph Learning
• Accelerating Backward Aggregation in GCN Training with Execution Path Preparing on GPUs
• On Self-Distilling Graph Neural Network
• Breaking the Expressive Bottlenecks of Graph Neural Networks
• Learning to Pool in Graph Neural Networks for Extrapolation
• Position-Sensing Graph Neural Networks: Proactively Learning Nodes Relative Positions
• Adaptive Neural Message Passing for Inductive Learning on Hypergraphs
• Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution
• LiteGEM: Lite Geometry Enhanced Molecular Representation Learning for Quantum Property Prediction
• On Local Aggregation in Heterophilic Graphs
• Signed Bipartite Graph Neural Networks
• Improving Graph Property Prediction with Generalized Readout Functions
• An FEA surrogate model with Boundary Oriented Graph Embedding approach
• Multipath Graph Convolutional Neural Networks
• Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks
• Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs
• Diversified Multiscale Graph Learning with Graph Self-Correction
• Pairwise Half-graph Discrimination: A Simple Graph-level Self-supervised Strategy for Pre-training Graph Neural Networks