Towards Deeper Graph Neural Networks with Differentiable Group Normalization

Abstract

Graph neural networks (GNNs), which learn the representation of a node by aggregating its neighbors, have become an effective computational tool in downstream applications. Over-smoothing is one of the key issues which limit the performance of GNNs as the number of layers increases. It is because the stacked aggregators would make node representations converge to indistinguishable vectors. Several attempts have been made to tackle the issue by bringing linked node pairs close and unlinked pairs distinct. However, they often ignore the intrinsic community structures and would result in sub-optimal performance. The representations of nodes within the same community/class need be similar to facilitate the classification, while different classes are expected to be separated in embedding space. To bridge the gap, we introduce two over-smoothing metrics and a novel technique, i.e., differentiable group normalization (DGN). It normalizes nodes within the same group independently to increase their smoothness, and separates node distributions among different groups to significantly alleviate the over-smoothing issue. Experiments on real-world datasets demonstrate that DGN makes GNN models more robust to over-smoothing and achieves better performance with deeper GNNs.

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Inductive Representation Learning on Large Graphs
• Graph Neural Networks: A Review of Methods and Applications
• Simplifying Graph Convolutional Networks
• Pitfalls of Graph Neural Network Evaluation
• Gated Graph Sequence Neural Networks
• Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
• How Powerful are Graph Neural Networks?
• Revisiting Semi-Supervised Learning with Graph Embeddings
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• PairNorm: Tackling Oversmoothing in GNNs
• Benchmarking Graph Neural Networks
• DeepGCNs: Can GCNs Go as Deep as CNNs?
• SpecAE: Spectral AutoEncoder for Anomaly Detection in Attributed Networks
• Multi-Channel Graph Convolutional Networks

Cited by in corpus (15)

• DeepGCNs: Making GCNs Go as Deep as CNNs
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Graph-less Collaborative Filtering
• Learning from the Dark: Boosting Graph Convolutional Neural Networks with Diverse Negative Samples
• Piezoelectric modulus prediction using machine learning and graph neural networks
• Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective
• Structure-Aware DropEdge Towards Deep Graph Convolutional Networks
• Evaluating Deep Graph Neural Networks
• UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks
• Towards Deepening Graph Neural Networks: A GNTK-based Optimization Perspective
• Learning Disentangled Representations for Time Series
• Jointly Learnable Data Augmentations for Self-Supervised GNNs
• Reversible and irreversible bracket-based dynamics for deep graph neural networks
• Normalisation and Initialisation Strategies for Graph Neural Networks in Blockchain Anomaly Detection
• Alleviating Over-Smoothing via Aggregation over Compact Manifolds