DeeperGCN: All You Need to Train Deeper GCNs

Abstract

Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. These challenges limit the representation power of GCNs on large-scale graphs. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs. We define differentiable generalized aggregation functions to unify different message aggregation operations (e.g. mean, max). We also propose a novel normalization layer namely MsgNorm and a pre-activation version of residual connections for GCNs. Extensive experiments on Open Graph Benchmark (OGB) show DeeperGCN significantly boosts performance over the state-of-the-art on the large scale graph learning tasks of node property prediction and graph property prediction. Please visit https://www.deepgcns.org for more information.

This work is still working in process. More results will be updated in the future version. Project website: https://www.deepgcns.org

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• Deep Convolutional Networks on Graph-Structured Data
• Fast Graph Representation Learning with PyTorch Geometric
• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
• Adaptive Graph Convolutional Neural Networks
• Geometric deep learning on graphs and manifolds using mixture model CNNs

Cited by in corpus (24)

• Variational Graph Normalized Auto-Encoders
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Label Information Enhanced Fraud Detection against Low Homophily in Graphs
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Nested Graph Neural Networks
• CGNN: Traffic Classification with Graph Neural Network
• Motif-Driven Contrastive Learning of Graph Representations
• Dual-view Molecule Pre-training
• Learnable Graph Matching: Incorporating Graph Partitioning with Deep Feature Learning for Multiple Object Tracking
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• On Graph Neural Networks versus Graph-Augmented MLPs
• LibAUC: A Deep Learning Library for X-Risk Optimization
• Molecule3D: A Benchmark for Predicting 3D Geometries from Molecular Graphs
• Fast Quantum Property Prediction via Deeper 2D and 3D Graph Networks
• Breaking the Expressive Bottlenecks of Graph Neural Networks
• Learning to Pool in Graph Neural Networks for Extrapolation
• Adaptive Neural Message Passing for Inductive Learning on Hypergraphs
• Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution
• LiteGEM: Lite Geometry Enhanced Molecular Representation Learning for Quantum Property Prediction
• On Local Aggregation in Heterophilic Graphs
• An FEA surrogate model with Boundary Oriented Graph Embedding approach
• Diversified Multiscale Graph Learning with Graph Self-Correction
• Signed Bipartite Graph Neural Networks
• Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks