Structure-Aware DropEdge Towards Deep Graph Convolutional Networks

Abstract

It has been discovered that Graph Convolutional Networks (GCNs) encounter a remarkable drop in performance when multiple layers are piled up. The main factor that accounts for why deep GCNs fail lies in over-smoothing, which isolates the network output from the input with the increase of network depth, weakening expressivity and trainability. In this paper, we start by investigating refined measures upon DropEdge -- an existing simple yet effective technique to relieve over-smoothing. We term our method as DropEdge++ for its two structure-aware samplers in contrast to DropEdge: layer-dependent sampler and feature-dependent sampler. Regarding the layer-dependent sampler, we interestingly find that increasingly sampling edges from the bottom layer yields superior performance than the decreasing counterpart as well as DropEdge. We theoretically reveal this phenomenon with Mean-Edge-Number (MEN), a metric closely related to over-smoothing. For the feature-dependent sampler, we associate the edge sampling probability with the feature similarity of node pairs, and prove that it further correlates the convergence subspace of the output layer with the input features. Extensive experiments on several node classification benchmarks, including both full- and semi- supervised tasks, illustrate the efficacy of DropEdge++ and its compatibility with a variety of backbones by achieving generally better performance over DropEdge and the no-drop version.

arXiv commentIEEE Transactions on Neural Networks and Learning Systems, 2023

References in corpus (19)

• A Comprehensive Survey on Graph Neural Networks
• DeepWalk: Online Learning of Social Representations
• Categorical Reparameterization with Gumbel-Softmax
• Spectral Networks and Locally Connected Networks on Graphs
• Deep Convolutional Networks on Graph-Structured Data
• Representation Learning on Graphs with Jumping Knowledge Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Pitfalls of Graph Neural Network Evaluation
• Large-Scale Learnable Graph Convolutional Networks
• Predict then Propagate: Graph Neural Networks meet Personalized PageRank
• Simple and Deep Graph Convolutional Networks
• GraphSAINT: Graph Sampling Based Inductive Learning Method
• PairNorm: Tackling Oversmoothing in GNNs
• Graph Neural Networks Exponentially Lose Expressive Power for Node Classification
• Diffusion Improves Graph Learning
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization
• Tackling Over-Smoothing for General Graph Convolutional Networks
• Quantifying the Alignment of Graph and Features in Deep Learning
• Graph Decoupling Attention Markov Networks for Semi-supervised Graph Node Classification