A Note on Over-Smoothing for Graph Neural Networks

Abstract

Graph Neural Networks (GNNs) have achieved a lot of success on graph-structured data. However, it is observed that the performance of graph neural networks does not improve as the number of layers increases. This effect, known as over-smoothing, has been analyzed mostly in linear cases. In this paper, we build upon previous results \cite{oono2019graph} to further analyze the over-smoothing effect in the general graph neural network architecture. We show when the weight matrix satisfies the conditions determined by the spectrum of augmented normalized Laplacian, the Dirichlet energy of embeddings will converge to zero, resulting in the loss of discriminative power. Using Dirichlet energy to measure "expressiveness" of embedding is conceptually clean; it leads to simpler proofs than \cite{oono2019graph} and can handle more non-linearities.

Appeared at ICML 2020 Graph Representation Learning workshop. https://grlplus.github.io/cfp/

References in corpus (11)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties
• Graph Neural Networks: A Review of Methods and Applications
• Simplifying Graph Convolutional Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning
• PairNorm: Tackling Oversmoothing in GNNs
• GMNN: Graph Markov Neural Networks
• DeepGCNs: Can GCNs Go as Deep as CNNs?
• Network Representation Learning: Consolidation and Renewed Bearing

Cited by in corpus (22)

• AdaGNN: Graph Neural Networks with Adaptive Frequency Response Filter
• Model Degradation Hinders Deep Graph Neural Networks
• A Heterogeneous Graph-Based Multi-Task Learning for Fault Event Diagnosis in Smart Grid
• Flow Completion Network: Inferring the Fluid Dynamics from Incomplete Flow Information using Graph Neural Networks
• Propagation Tree Is Not Deep: Adaptive Graph Contrastive Learning Approach for Rumor Detection
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• On the design space between molecular mechanics and machine learning force fields
• Deep learning for dynamic graphs: models and benchmarks
• Convolutional Learning on Multigraphs
• Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures
• On Provable Benefits of Depth in Training Graph Convolutional Networks
• Advancing Molecular Machine Learning Representations with Stereoelectronics-Infused Molecular Graphs
• Evaluating Deep Graph Neural Networks
• Revisiting Graph Convolutional Network on Semi-Supervised Node Classification from an Optimization Perspective
• Efficient Anatomical Labeling of Pulmonary Tree Structures via Deep Point-Graph Representation-based Implicit Fields
• Stochastic Aggregation in Graph Neural Networks
• Graph Coordinates and Conventional Neural Networks -- An Alternative for Graph Neural Networks
• Local, global and scale-dependent node roles
• SoGCN: Second-Order Graph Convolutional Networks
• AViTMP: A Tracking-Specific Transformer for Single-Branch Visual Tracking
• Reversible and irreversible bracket-based dynamics for deep graph neural networks
• Graph Partner Neural Networks for Semi-Supervised Learning on Graphs