Pitfalls of Graph Neural Network Evaluation

Abstract

Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.

References in corpus (3)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Adversarial Attacks on Neural Networks for Graph Data

Cited by in corpus (183)

• Graph Neural Networks: A Review of Methods and Applications
• Fast Graph Representation Learning with PyTorch Geometric
• Graph Contrastive Learning with Adaptive Augmentation
• Graph Self-Supervised Learning: A Survey
• Towards Deeper Graph Neural Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Self-Attention Graph Pooling
• A Gentle Introduction to Deep Learning for Graphs
• PairNorm: Tackling Oversmoothing in GNNs
• Benchmarking Graph Neural Networks
• SIGN: Scalable Inception Graph Neural Networks
• Beyond Homophily in Graph Neural Networks: Current Limitations and Effective Designs
• Design Space for Graph Neural Networks
• Deep Learning on Graphs: A Survey
• Graph Neural Networks: Taxonomy, Advances and Trends
• Learning on Attribute-Missing Graphs
• A Fair Comparison of Graph Neural Networks for Graph Classification
• How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
• Graph Attention Multi-Layer Perceptron
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks
• Unifying Graph Convolutional Neural Networks and Label Propagation
• Edge Contraction Pooling for Graph Neural Networks
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Adaptive Universal Generalized PageRank Graph Neural Network
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability
• Variational Graph Normalized Auto-Encoders
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization
• Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological View
• Subgraph Federated Learning with Missing Neighbor Generation
• Automated Machine Learning on Graphs: A Survey
• Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery
• Directed Graph Convolutional Network
• A Comparative Study for Unsupervised Network Representation Learning
• Graph Communal Contrastive Learning
• Rethinking pooling in graph neural networks
• Uncertainty Aware Semi-Supervised Learning on Graph Data
• Graph Clustering with Graph Neural Networks
• Graph InfoClust: Leveraging cluster-level node information for unsupervised graph representation learning
• Topology-Imbalance Learning for Semi-Supervised Node Classification
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation
• Learning Strong Graph Neural Networks with Weak Information
• Predicting Biomedical Interactions with Higher-Order Graph Convolutional Networks
• CGC: Contrastive Graph Clustering for Community Detection and Tracking
• Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
• CLARE: A Semi-supervised Community Detection Algorithm
• Relational Pooling for Graph Representations
• GraphFL: A Federated Learning Framework for Semi-Supervised Node Classification on Graphs
• Understanding over-squashing and bottlenecks on graphs via curvature
• Space4HGNN: A Novel, Modularized and Reproducible Platform to Evaluate Heterogeneous Graph Neural Network
• Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective
• Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks
• Graph Neural Networks: Methods, Applications, and Opportunities
• AdaGCN: Adaboosting Graph Convolutional Networks into Deep Models
• Large-Scale Representation Learning on Graphs via Bootstrapping
• Learning on Graphs with Out-of-Distribution Nodes
• Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective
• Subgroup Generalization and Fairness of Graph Neural Networks
• Just Jump: Dynamic Neighborhood Aggregation in Graph Neural Networks
• New Benchmarks for Learning on Non-Homophilous Graphs
• GraphWorld: Fake Graphs Bring Real Insights for GNNs
• Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks
• Graph Policy Network for Transferable Active Learning on Graphs
• ROD: Reception-aware Online Distillation for Sparse Graphs
• Graph Random Neural Network for Semi-Supervised Learning on Graphs
• Tripartite Heterogeneous Graph Propagation for Large-scale Social Recommendation
• Automated Self-Supervised Learning for Graphs
• Structure-Aware DropEdge Towards Deep Graph Convolutional Networks
• Decimated Framelet System on Graphs and Fast G-Framelet Transforms
• Graph Neural Networks with Diverse Spectral Filtering
• Hyperbolic Geometric Graph Representation Learning for Hierarchy-imbalance Node Classification
• HeteGCN: Heterogeneous Graph Convolutional Networks for Text Classification
• Local Augmentation for Graph Neural Networks
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Learned Low Precision Graph Neural Networks
• Multi-Scale Contrastive Siamese Networks for Self-Supervised Graph Representation Learning
• Structack: Structure-based Adversarial Attacks on Graph Neural Networks
• DEMO-Net: Degree-specific Graph Neural Networks for Node and Graph Classification
• Beltrami Flow and Neural Diffusion on Graphs
• Residual Network and Embedding Usage: New Tricks of Node Classification with Graph Convolutional Networks
• Single-Node Attacks for Fooling Graph Neural Networks
• A Large-Scale Database for Graph Representation Learning
• Large-scale graph representation learning with very deep GNNs and self-supervision
• A Hybrid Similarity-Aware Graph Neural Network with Transformer for Node Classification
• Reliable Node Similarity Matrix Guided Contrastive Graph Clustering
• Explicit Feature Interaction-aware Graph Neural Networks
• NODE-SELECT: A Graph Neural Network Based On A Selective Propagation Technique
• Meta-Learning with Graph Neural Networks: Methods and Applications
• p2pGNN: A Decentralized Graph Neural Network for Node Classification in Peer-to-Peer Networks
• Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning
• FL-AGCNS: Federated Learning Framework for Automatic Graph Convolutional Network Search
• Ego-GNNs: Exploiting Ego Structures in Graph Neural Networks
• Fisher-Bures Adversary Graph Convolutional Networks
• Hyperbolic Deep Neural Networks: A Survey
• Robust Node Classification on Graphs: Jointly from Bayesian Label Transition and Topology-based Label Propagation
• Self-supervised Graph Neural Networks without explicit negative sampling
• On the Unreasonable Effectiveness of Feature propagation in Learning on Graphs with Missing Node Features
• On the Equivalence of Decoupled Graph Convolution Network and Label Propagation
• Distance-wise Prototypical Graph Neural Network in Node Imbalance Classification
• SCE: Scalable Network Embedding from Sparsest Cut
• BSAL: A Framework of Bi-component Structure and Attribute Learning for Link Prediction
• Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classification
• Graph Convolutional Gaussian Processes For Link Prediction
• Self-Enhanced GNN: Improving Graph Neural Networks Using Model Outputs
• Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework
• MathNet: Haar-Like Wavelet Multiresolution-Analysis for Graph Representation and Learning
• SNoRe: Scalable Unsupervised Learning of Symbolic Node Representations
• Are Graph Neural Networks Miscalibrated?
• Label-GCN: An Effective Method for Adding Label Propagation to Graph Convolutional Networks
• Node Masking: Making Graph Neural Networks Generalize and Scale Better
• Graph-Revised Convolutional Network
• Modeling multi-scale data via a network of networks
• Clarify Confused Nodes via Separated Learning
• LGD-GCN: Local and Global Disentangled Graph Convolutional Networks
• Calibrating and Improving Graph Contrastive Learning
• Sparse Graph Attention Networks
• Topological based classification using graph convolutional networks
• A pipeline for fair comparison of graph neural networks in node classification tasks
• CogDL: A Comprehensive Library for Graph Deep Learning
• Metapath- and Entity-aware Graph Neural Network for Recommendation
• Neural Architecture Search in Graph Neural Networks
• Semi-supervised Anomaly Detection on Attributed Graphs
• Adversarial Graph Disentanglement
• Node2Seq: Towards Trainable Convolutions in Graph Neural Networks
• Negative-Free Self-Supervised Gaussian Embedding of Graphs
• Node Classification Meets Link Prediction on Knowledge Graphs
• PushNet: Efficient and Adaptive Neural Message Passing
• Spectral Graph Attention Network with Fast Eigen-approximation
• Data Augmentation View on Graph Convolutional Network and the Proposal of Monte Carlo Graph Learning
• $p$-Laplacian Based Graph Neural Networks
• Robustness of Graph Neural Networks at Scale
• Node Copying: A Random Graph Model for Effective Graph Sampling
• Image Retrieval for Structure-from-Motion via Graph Convolutional Network
• HighwayGraph: Modelling Long-distance Node Relations for Improving General Graph Neural Network
• Evaluating Modules in Graph Contrastive Learning
• Releasing Graph Neural Networks with Differential Privacy Guarantees
• Search For Deep Graph Neural Networks
• Scalable Deep Graph Clustering with Random-walk based Self-supervised Learning
• An Empirical Study of Graph Contrastive Learning
• Graph Belief Propagation Networks
• Edgeless-GNN: Unsupervised Representation Learning for Edgeless Nodes
• Local2Global: Scaling global representation learning on graphs via local training
• BiGCN: A Bi-directional Low-Pass Filtering Graph Neural Network
• Multi-view Contrastive Graph Clustering
• Stacked Graph Filter
• On the Impact of Communities on Semi-supervised Classification Using Graph Neural Networks
• GLAM: Graph Learning by Modeling Affinity to Labeled Nodes for Graph Neural Networks
• EWasteNet: A Two-Stream Data Efficient Image Transformer Approach for E-Waste Classification
• Should Graph Neural Networks Use Features, Edges, Or Both?
• Neuron with Steady Response Leads to Better Generalization
• Towards Time-Aware Context-Aware Deep Trust Prediction in Online Social Networks
• Graph Neural Networks with Feature and Structure Aware Random Walk
• Structure-Aware Label Smoothing for Graph Neural Networks
• GmCN: Graph Mask Convolutional Network
• Neural Trees for Learning on Graphs
• Distribution Matching for Graph Quantification Under Structural Covariate Shift
• Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representations
• Learning Conjoint Attentions for Graph Neural Nets
• Context-Aware Graph Attention Networks
• Lorentzian Graph Convolutional Networks
• Graphs, Entities, and Step Mixture
• Jointly Learnable Data Augmentations for Self-Supervised GNNs
• Reversible and irreversible bracket-based dynamics for deep graph neural networks
• Deep Constraint-based Propagation in Graph Neural Networks
• On Local Aggregation in Heterophilic Graphs
• A Process for the Evaluation of Node Embedding Methods in the Context of Node Classification
• Node Feature Kernels Increase Graph Convolutional Network Robustness
• Scaling Up Graph Neural Networks Via Graph Coarsening
• Multipath Graph Convolutional Neural Networks
• Graph Partner Neural Networks for Semi-Supervised Learning on Graphs
• Tracing the Propagation Path: A Flow Perspective of Representation Learning on Graphs
• Towards Quantized Model Parallelism for Graph-Augmented MLPs Based on Gradient-Free ADMM Framework
• Data-Driven Self-Supervised Graph Representation Learning
• Classifying Diagrams and Their Parts using Graph Neural Networks: A Comparison of Crowd-Sourced and Expert Annotations
• Curvature Graph Neural Network
• Graphs for deep learning representations
• Towards a Taxonomy of Graph Learning Datasets
• Which Samples Should be Learned First: Easy or Hard?
• Co-embedding of Nodes and Edges with Graph Neural Networks
• Does your graph need a confidence boost? Convergent boosted smoothing on graphs with tabular node features
• Contributions to Representation Learning with Graph Autoencoders and Applications to Music Recommendation