Dirichlet Energy Constrained Learning for Deep Graph Neural Networks

Abstract

Graph neural networks (GNNs) integrate deep architectures and topological structure modeling in an effective way. However, the performance of existing GNNs would decrease significantly when they stack many layers, because of the over-smoothing issue. Node embeddings tend to converge to similar vectors when GNNs keep recursively aggregating the representations of neighbors. To enable deep GNNs, several methods have been explored recently. But they are developed from either techniques in convolutional neural networks or heuristic strategies. There is no generalizable and theoretical principle to guide the design of deep GNNs. To this end, we analyze the bottleneck of deep GNNs by leveraging the Dirichlet energy of node embeddings, and propose a generalizable principle to guide the training of deep GNNs. Based on it, a novel deep GNN framework -- EGNN is designed. It could provide lower and upper constraints in terms of Dirichlet energy at each layer to avoid over-smoothing. Experimental results demonstrate that EGNN achieves state-of-the-art performance by using deep layers.

References in corpus (27)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Fast Graph Representation Learning with PyTorch Geometric
• Simplifying Graph Convolutional Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• Pitfalls of Graph Neural Network Evaluation
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Predicting multicellular function through multi-layer tissue networks
• Towards Deeper Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Gated Graph Sequence Neural Networks
• How Powerful are Graph Neural Networks?
• Revisiting Semi-Supervised Learning with Graph Embeddings
• Link Prediction Based on Graph Neural Networks
• DeeperGCN: All You Need to Train Deeper GCNs
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• PairNorm: Tackling Oversmoothing in GNNs
• A Note on Over-Smoothing for Graph Neural Networks
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• On the Effects of Batch and Weight Normalization in Generative Adversarial Networks
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• Tackling Over-Smoothing for General Graph Convolutional Networks
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Deep Isometric Learning for Visual Recognition
• Multi-Channel Graph Convolutional Networks

Cited by in corpus (6)

• A variational quantum algorithm for the Feynman-Kac formula
• Beyond Over-smoothing: Uncovering the Trainability Challenges in Deep Graph Neural Networks
• COSCO: A Sharpness-Aware Training Framework for Few-shot Multivariate Time Series Classification
• CAP: Co-Adversarial Perturbation on Weights and Features for Improving Generalization of Graph Neural Networks
• DeepHGCN: Toward Deeper Hyperbolic Graph Convolutional Networks
• Graph Partner Neural Networks for Semi-Supervised Learning on Graphs