Tackling Over-Smoothing for General Graph Convolutional Networks

Abstract

Increasing the depth of GCN, which is expected to permit more expressivity, is shown to incur performance detriment especially on node classification. The main cause of this lies in over-smoothing. The over-smoothing issue drives the output of GCN towards a space that contains limited distinguished information among nodes, leading to poor expressivity. Several works on refining the architecture of deep GCN have been proposed, but it is still unknown in theory whether or not these refinements are able to relieve over-smoothing. In this paper, we first theoretically analyze how general GCNs act with the increase in depth, including generic GCN, GCN with bias, ResGCN, and APPNP. We find that all these models are characterized by a universal process: all nodes converging to a cuboid. Upon this theorem, we propose DropEdge to alleviate over-smoothing by randomly removing a certain number of edges at each training epoch. Theoretically, DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by dimension collapse. Experimental evaluations on simulated dataset have visualized the difference in over-smoothing between different GCNs. Moreover, extensive experiments on several real benchmarks support that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs.

arXiv commentSubmitted to TPAMI, 15 pages

References in corpus (10)

• Improving neural networks by preventing co-adaptation of feature detectors
• Inductive Representation Learning on Large Graphs
• Deep Convolutional Networks on Graph-Structured Data
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• How Powerful are Graph Neural Networks?
• PairNorm: Tackling Oversmoothing in GNNs
• Adaptive Sampling Towards Fast Graph Representation Learning
• Understanding the Representation Power of Graph Neural Networks in Learning Graph Topology
• Query2box: Reasoning over Knowledge Graphs in Vector Space using Box Embeddings

Cited by in corpus (12)

• Self-Supervised Graph Transformer on Large-Scale Molecular Data
• MGDCF: Distance Learning via Markov Graph Diffusion for Neural Collaborative Filtering
• Flow Completion Network: Inferring the Fluid Dynamics from Incomplete Flow Information using Graph Neural Networks
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• Structure-Aware DropEdge Towards Deep Graph Convolutional Networks
• On Provable Benefits of Depth in Training Graph Convolutional Networks
• Deep Graph Neural Networks with Shallow Subgraph Samplers
• DGDNN: Decoupled Graph Diffusion Neural Network for Stock Movement Prediction
• Contrastive Adaptive Propagation Graph Neural Networks for Efficient Graph Learning
• Multi-Level Attention Pooling for Graph Neural Networks: Unifying Graph Representations with Multiple Localities
• Alleviating Over-Smoothing via Aggregation over Compact Manifolds
• Interferometric Graph Transform for Community Labeling