PairNorm: Tackling Oversmoothing in GNNs

Abstract

The performance of graph neural nets (GNNs) is known to gradually decrease with increasing number of layers. This decay is partly attributed to oversmoothing, where repeated graph convolutions eventually make node embeddings indistinguishable. We take a closer look at two different interpretations, aiming to quantify oversmoothing. Our main contribution is PairNorm, a novel normalization layer that is based on a careful analysis of the graph convolution operator, which prevents all node embeddings from becoming too similar. What is more, PairNorm is fast, easy to implement without any change to network architecture nor any additional parameters, and is broadly applicable to any GNN. Experiments on real-world graphs demonstrate that PairNorm makes deeper GCN, GAT, and SGC models more robust against oversmoothing, and significantly boosts performance for a new problem setting that benefits from deeper GNNs. Code is available at https://github.com/LingxiaoShawn/PairNorm.

arXiv commentICLR 2020 Camera Ready

References in corpus (7)

• Inductive Representation Learning on Large Graphs
• Simplifying Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• Pitfalls of Graph Neural Network Evaluation
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• GMNN: Graph Markov Neural Networks
• Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks

Cited by in corpus (68)

• Principal Neighbourhood Aggregation for Graph Nets
• SIGN: Scalable Inception Graph Neural Networks
• Graph Neural Networks: Taxonomy, Advances and Trends
• Differentiable Graph Module (DGM) for Graph Convolutional Networks
• Semi-supervised Credit Card Fraud Detection via Attribute-Driven Graph Representation
• Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction
• A Note on Over-Smoothing for Graph Neural Networks
• Adaptive Universal Generalized PageRank Graph Neural Network
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• Variational Graph Normalized Auto-Encoders
• Graph Unrolling Networks: Interpretable Neural Networks for Graph Signal Denoising
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
• DropMessage: Unifying Random Dropping for Graph Neural Networks
• Tackling Over-Smoothing for General Graph Convolutional Networks
• Implicit Graph Neural Networks
• Revisiting Over-smoothing in Deep GCNs
• AdaGNN: Graph Neural Networks with Adaptive Frequency Response Filter
• Scalable Graph Neural Networks via Bidirectional Propagation
• Model Degradation Hinders Deep Graph Neural Networks
• How Attentive are Graph Attention Networks?
• EEG-GCNN: Augmenting Electroencephalogram-based Neurological Disease Diagnosis using a Domain-guided Graph Convolutional Neural Network
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Bridging the Gap between Spatial and Spectral Domains: A Survey on Graph Neural Networks
• Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective
• Beyond Low-Pass Filters: Adaptive Feature Propagation on Graphs
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Graph Random Neural Network for Semi-Supervised Learning on Graphs
• Gophormer: Ego-Graph Transformer for Node Classification
• Structure-Aware DropEdge Towards Deep Graph Convolutional Networks
• Pointer Graph Networks
• Dissecting the Diffusion Process in Linear Graph Convolutional Networks
• Goal-Aware Neural SAT Solver
• Learning to Drop: Robust Graph Neural Network via Topological Denoising
• Residual Network and Embedding Usage: New Tricks of Node Classification with Graph Convolutional Networks
• Deep Graph Neural Networks with Shallow Subgraph Samplers
• SSFG: Stochastically Scaling Features and Gradients for Regularizing Graph Convolutional Networks
• NODE-SELECT: A Graph Neural Network Based On A Selective Propagation Technique
• Ripple Walk Training: A Subgraph-based training framework for Large and Deep Graph Neural Network
• Learning Graph Normalization for Graph Neural Networks
• Activity-aware Human Mobility Prediction with Hierarchical Graph Attention Recurrent Network
• You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
• Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
• From Local Structures to Size Generalization in Graph Neural Networks
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• Finding Heterophilic Neighbors via Confidence-based Subgraph Matching for Semi-supervised Node Classification
• Sequential Graph Convolutional Network for Active Learning
• Scalar Field Prediction on Meshes Using Interpolated Multi-Resolution Convolutional Neural Networks
• Optimization and Generalization Analysis of Transduction through Gradient Boosting and Application to Multi-scale Graph Neural Networks
• Lipschitz Normalization for Self-Attention Layers with Application to Graph Neural Networks
• A pipeline for fair comparison of graph neural networks in node classification tasks
• DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability
• Maximum Entropy Weighted Independent Set Pooling for Graph Neural Networks
• Tree Structure-Aware Graph Representation Learning via Integrated Hierarchical Aggregation and Relational Metric Learning
• HighwayGraph: Modelling Long-distance Node Relations for Improving General Graph Neural Network
• Segmentation of EM showers for neutrino experiments with deep graph neural networks
• AKE-GNN: Effective Graph Learning with Adaptive Knowledge Exchange
• BiTe-GCN: A New GCN Architecture via BidirectionalConvolution of Topology and Features on Text-Rich Networks
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities
• Connecting Graph Convolutional Networks and Graph-Regularized PCA
• Network representation learning: A macro and micro view
• FeatureNorm: L2 Feature Normalization for Dynamic Graph Embedding
• Compensation Learning
• AN-GCN: An Anonymous Graph Convolutional Network Defense Against Edge-Perturbing Attack
• Multipath Graph Convolutional Neural Networks
• Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework