Benchmarking Graph Neural Networks

Abstract

In the last few years, graph neural networks (GNNs) have become the standard toolkit for analyzing and learning from data on graphs. This emerging field has witnessed an extensive growth of promising techniques that have been applied with success to computer science, mathematics, biology, physics and chemistry. But for any successful field to become mainstream and reliable, benchmarks must be developed to quantify progress. This led us in March 2020 to release a benchmark framework that i) comprises of a diverse collection of mathematical and real-world graphs, ii) enables fair model comparison with the same parameter budget to identify key architectures, iii) has an open-source, easy-to-use and reproducible code infrastructure, and iv) is flexible for researchers to experiment with new theoretical ideas. As of December 2022, the GitHub repository has reached 2,000 stars and 380 forks, which demonstrates the utility of the proposed open-source framework through the wide usage by the GNN community. In this paper, we present an updated version of our benchmark with a concise presentation of the aforementioned framework characteristics, an additional medium-sized molecular dataset AQSOL, similar to the popular ZINC, but with a real-world measured chemical target, and discuss how this framework can be leveraged to explore new GNN designs and insights. As a proof of value of our benchmark, we study the case of graph positional encoding (PE) in GNNs, which was introduced with this benchmark and has since spurred interest of exploring more powerful PE for Transformers and GNNs in a robust experimental setting.

arXiv commentBenchmarking framework on GitHub at https://github.com/graphdeeplearning/benchmarking-gnns

References in corpus (39)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• Spectral Networks and Locally Connected Networks on Graphs
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Fast Graph Representation Learning with PyTorch Geometric
• Learning Combinatorial Optimization Algorithms over Graphs
• Representation Learning on Graphs with Jumping Knowledge Networks
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Pitfalls of Graph Neural Network Evaluation
• Interaction Networks for Learning about Objects, Relations and Physics
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Learning to Simulate Complex Physics with Graph Networks
• Self-Attention Graph Pooling
• A Gentle Introduction to Deep Learning for Graphs
• Combinatorial Optimization with Graph Convolutional Networks and Guided Tree Search
• Revisiting Graph Neural Networks: All We Have is Low-Pass Filters
• Residual Gated Graph ConvNets
• DeepGCNs: Making GCNs Go as Deep as CNNs
• An Efficient Graph Convolutional Network Technique for the Travelling Salesman Problem
• Fake News Detection on Social Media using Geometric Deep Learning
• SIGN: Scalable Inception Graph Neural Networks
• Attention, Learn to Solve Routing Problems!
• Chip Placement with Deep Reinforcement Learning
• A Fair Comparison of Graph Neural Networks for Graph Classification
• Invariant and Equivariant Graph Networks
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Position-aware Graph Neural Networks
• A Survey of Deep Learning for Scientific Discovery
• Understanding Attention and Generalization in Graph Neural Networks
• A Two-Step Graph Convolutional Decoder for Molecule Generation
• Are Powerful Graph Neural Nets Necessary? A Dissection on Graph Classification
• GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation
• Learning Symbolic Physics with Graph Networks
• Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling
• On Graph Classification Networks, Datasets and Baselines
• Redundancy-Free Computation Graphs for Graph Neural Networks

Cited by in corpus (114)

• Graph Contrastive Learning with Augmentations
• Learning Intents behind Interactions with Knowledge Graph for Recommendation
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• A Generalization of Transformer Networks to Graphs
• TUDataset: A collection of benchmark datasets for learning with graphs
• Improving Graph Neural Network Expressivity via Subgraph Isomorphism Counting
• Principal Neighbourhood Aggregation for Graph Nets
• DeepGCNs: Making GCNs Go as Deep as CNNs
• Design Space for Graph Neural Networks
• Graph Neural Networks: Taxonomy, Advances and Trends
• Adversarial Graph Augmentation to Improve Graph Contrastive Learning
• Do Transformers Really Perform Bad for Graph Representation?
• A Survey on Graph-Based Deep Learning for Computational Histopathology
• Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification
• Machine Learning on Graphs: A Model and Comprehensive Taxonomy
• Towards Deeper Graph Neural Networks with Differentiable Group Normalization
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Rethinking Graph Transformers with Spectral Attention
• Factorizable Graph Convolutional Networks
• Automated Machine Learning on Graphs: A Survey
• Accurate Learning of Graph Representations with Graph Multiset Pooling
• Understanding the Performance of Knowledge Graph Embeddings in Drug Discovery
• On Representation Knowledge Distillation for Graph Neural Networks
• Graph Kernels: State-of-the-Art and Future Challenges
• Rethinking pooling in graph neural networks
• Graph Clustering with Graph Neural Networks
• Beyond Low-pass Filtering: Graph Convolutional Networks with Automatic Filtering
• How Attentive are Graph Attention Networks?
• PGM-Explainer: Probabilistic Graphical Model Explanations for Graph Neural Networks
• Graph Neural Networks for Graph Drawing
• Learning the Travelling Salesperson Problem Requires Rethinking Generalization
• Stealing Links from Graph Neural Networks
• Hierarchical Inter-Message Passing for Learning on Molecular Graphs
• Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks
• Graph Neural Networks: Methods, Applications, and Opportunities
• Heterogeneous Network Representation Learning: A Unified Framework with Survey and Benchmark
• Finding Patient Zero: Learning Contagion Source with Graph Neural Networks
• Can Graph Neural Networks Count Substructures?
• Graph Deep Learning for Time Series Forecasting
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awareness
• Lifelong Learning of Graph Neural Networks for Open-World Node Classification
• New Benchmarks for Learning on Non-Homophilous Graphs
• Graph Neural Networks with Learnable Structural and Positional Representations
• Expressive Power of Invariant and Equivariant Graph Neural Networks
• Building powerful and equivariant graph neural networks with structural message-passing
• Visualizing Graph Neural Networks with CorGIE: Corresponding a Graph to Its Embedding
• Degree-Quant: Quantization-Aware Training for Graph Neural Networks
• Edge Representation Learning with Hypergraphs
• Pointer Graph Networks
• Beyond Graph Neural Networks with Lifted Relational Neural Networks
• Spatial Aggregation and Temporal Convolution Networks for Real-time Kriging
• Erdos Goes Neural: an Unsupervised Learning Framework for Combinatorial Optimization on Graphs
• Large-scale graph representation learning with very deep GNNs and self-supervision
• Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
• Equivariant Subgraph Aggregation Networks
• Learning Graph Normalization for Graph Neural Networks
• Ego-GNNs: Exploiting Ego Structures in Graph Neural Networks
• Global Attention Improves Graph Networks Generalization
• Geometric Scattering Attention Networks
• Evaluating Curriculum Learning Strategies in Neural Combinatorial Optimization
• HistoCartography: A Toolkit for Graph Analytics in Digital Pathology
• Directional Graph Networks
• Graph Convolutional Networks for traffic anomaly
• From Local Structures to Size Generalization in Graph Neural Networks
• You are AllSet: A Multiset Function Framework for Hypergraph Neural Networks
• Reconstructing nodal pressures in water distribution systems with graph neural networks
• MathNet: Haar-Like Wavelet Multiresolution-Analysis for Graph Representation and Learning
• Rethinking Graph Neural Architecture Search from Message-passing
• Neural Sentence Ordering Based on Constraint Graphs
• Adversarial Attacks on Graph Classification via Bayesian Optimisation
• Lymph Node Graph Neural Networks for Cancer Metastasis Prediction
• Relational Graph Learning on Visual and Kinematics Embeddings for Accurate Gesture Recognition in Robotic Surgery
• CogDL: A Comprehensive Library for Graph Deep Learning
• Weisfeiler and Lehman Go Cellular: CW Networks
• A pipeline for fair comparison of graph neural networks in node classification tasks
• GCN-SL: Graph Convolutional Networks with Structure Learning for Graphs under Heterophily
• Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation
• Pose-GNN : Camera Pose Estimation System Using Graph Neural Networks
• Maximum Entropy Weighted Independent Set Pooling for Graph Neural Networks
• Reconstruction for Powerful Graph Representations
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition
• Evaluating Modules in Graph Contrastive Learning
• Breaking the Expressive Bottlenecks of Graph Neural Networks
• On Self-Distilling Graph Neural Network
• Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning
• Parameter Prediction for Unseen Deep Architectures
• Should Graph Neural Networks Use Features, Edges, Or Both?
• Graph Computing for Financial Crime and Fraud Detection: Trends, Challenges and Outlook
• Amortized Probabilistic Detection of Communities in Graphs
• From Spectrum Wavelet to Vertex Propagation: Graph Convolutional Networks Based on Taylor Approximation
• Efficient Colon Cancer Grading with Graph Neural Networks
• Message Passing in Graph Convolution Networks via Adaptive Filter Banks
• SoGCN: Second-Order Graph Convolutional Networks
• Topological Graph Neural Networks
• Partition and Code: learning how to compress graphs
• Feature Correlation Aggregation: on the Path to Better Graph Neural Networks
• Learning Conjoint Attentions for Graph Neural Nets
• Learning Parametrised Graph Shift Operators
• Deep Constraint-based Propagation in Graph Neural Networks
• On the Importance of Sampling in Training GCNs: Tighter Analysis and Variance Reduction
• Fea2Fea: Exploring Structural Feature Correlations via Graph Neural Networks
• A Graph Neural Network Approach for Scalable and Dynamic IP Similarity in Enterprise Networks
• Graph2Graph Learning with Conditional Autoregressive Models
• Edge but not Least: Cross-View Graph Pooling
• Meta-Aggregator: Learning to Aggregate for 1-bit Graph Neural Networks
• The Role of Isomorphism Classes in Multi-Relational Datasets
• Edge-featured Graph Neural Architecture Search
• Graph Neural Network for Hamiltonian-Based Material Property Prediction
• An Experimental Study of the Transferability of Spectral Graph Networks
• LSP : Acceleration and Regularization of Graph Neural Networks via Locality Sensitive Pruning of Graphs
• Symmetry-driven graph neural networks
• Scaling up graph homomorphism for classification via sampling