Revisiting Over-smoothing in Deep GCNs

Abstract

Oversmoothing has been assumed to be the major cause of performance drop in deep graph convolutional networks (GCNs). In this paper, we propose a new view that deep GCNs can actually learn to anti-oversmooth during training. This work interprets a standard GCN architecture as layerwise integration of a Multi-layer Perceptron (MLP) and graph regularization. We analyze and conclude that before training, the final representation of a deep GCN does over-smooth, however, it learns anti-oversmoothing during training. Based on the conclusion, the paper further designs a cheap but effective trick to improve GCN training. We verify our conclusions and evaluate the trick on three citation networks and further provide insights on neighborhood aggregation in GCNs.

arXiv comment19 pages

References in corpus (19)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Discrete Signal Processing on Graphs
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Simplifying Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• Discrete Signal Processing on Graphs: Sampling Theory
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
• How Powerful are Graph Neural Networks?
• Revisiting Semi-Supervised Learning with Graph Embeddings
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• PairNorm: Tackling Oversmoothing in GNNs
• SpectralNet: Spectral Clustering using Deep Neural Networks
• Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology
• Hyperbolic Graph Neural Networks
• Deep Iterative and Adaptive Learning for Graph Neural Networks
• What graph neural networks cannot learn: depth vs width

Cited by in corpus (13)

• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• GATE: Graph CCA for Temporal SElf-supervised Learning for Label-efficient fMRI Analysis
• Flow Completion Network: Inferring the Fluid Dynamics from Incomplete Flow Information using Graph Neural Networks
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• On Provable Benefits of Depth in Training Graph Convolutional Networks
• Evaluating Deep Graph Neural Networks
• Node Similarity Preserving Graph Convolutional Networks
• SSFG: Stochastically Scaling Features and Gradients for Regularizing Graph Convolutional Networks
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoods
• Improving the Long-Range Performance of Gated Graph Neural Networks
• DyDiff-VAE: A Dynamic Variational Framework for Information Diffusion Prediction
• An FEA surrogate model with Boundary Oriented Graph Embedding approach