Design Space for Graph Neural Networks

Abstract

The rapid evolution of Graph Neural Networks (GNNs) has led to a growing number of new architectures as well as novel applications. However, current research focuses on proposing and evaluating specific architectural designs of GNNs, as opposed to studying the more general design space of GNNs that consists of a Cartesian product of different design dimensions, such as the number of layers or the type of the aggregation function. Additionally, GNN designs are often specialized to a single task, yet few efforts have been made to understand how to quickly find the best GNN design for a novel task or a novel dataset. Here we define and systematically study the architectural design space for GNNs which consists of 315,000 different designs over 32 different predictive tasks. Our approach features three key innovations: (1) A general GNN design space; (2) a GNN task space with a similarity metric, so that for a given novel task/dataset, we can quickly identify/transfer the best performing architecture; (3) an efficient and effective design space evaluation method which allows insights to be distilled from a huge number of model-task combinations. Our key results include: (1) A comprehensive set of guidelines for designing well-performing GNNs; (2) while best GNN designs for different tasks vary significantly, the GNN task space allows for transferring the best designs across different tasks; (3) models discovered using our design space achieve state-of-the-art performance. Overall, our work offers a principled and scalable approach to transition from studying individual GNN designs for specific tasks, to systematically studying the GNN design space and the task space. Finally, we release GraphGym, a powerful platform for exploring different GNN designs and tasks. GraphGym features modularized GNN implementation, standardized GNN evaluation, and reproducible and scalable experiment management.

arXiv commentNeurIPS 2020 (Spotlight). Typos fixed

References in corpus (21)

• Inductive Representation Learning on Large Graphs
• Growing Scale-Free Networks with Tunable Clustering
• Representation Learning on Graphs with Jumping Knowledge Networks
• Variational Graph Auto-Encoders
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Searching for Activation Functions
• Pitfalls of Graph Neural Network Evaluation
• Predicting multicellular function through multi-layer tissue networks
• Benchmarking Graph Neural Networks
• A Fair Comparison of Graph Neural Networks for Graph Classification
• DeepGCNs: Can GCNs Go as Deep as CNNs?
• Handling Missing Data with Graph Representation Learning
• Position-aware Graph Neural Networks
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• Transfer Learning with Neural AutoML
• Neural Subgraph Matching
• Meta Architecture Search
• Hierarchical Temporal Convolutional Networks for Dynamic Recommender Systems
• Graph Structure of Neural Networks
• G2SAT: Learning to Generate SAT Formulas

Cited by in corpus (32)

• Understanding Pooling in Graph Neural Networks
• Handling Missing Data with Graph Representation Learning
• Automated Machine Learning on Graphs: A Survey
• Power Flow Balancing with Decentralized Graph Neural Networks
• DiffMG: Differentiable Meta Graph Search for Heterogeneous Graph Neural Networks
• Graph-MLP: Node Classification without Message Passing in Graph
• PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm
• Predicting Basin Stability of Power Grids using Graph Neural Networks
• Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration
• Learning Graph Neural Networks with Positive and Unlabeled Nodes
• Space4HGNN: A Novel, Modularized and Reproducible Platform to Evaluate Heterogeneous Graph Neural Network
• Designing the Topology of Graph Neural Networks: A Novel Feature Fusion Perspective
• Graph Neural Networks: Methods, Applications, and Opportunities
• GraphWorld: Fake Graphs Bring Real Insights for GNNs
• IGNNITION: Bridging the Gap Between Graph Neural Networks and Networking Systems
• Anomaly Detection with Score Distribution Discrimination
• Search to Capture Long-range Dependency with Stacking GNNs for Graph Classification
• Learned Low Precision Graph Neural Networks
• A graph neural network-based model with Out-of-Distribution Robustness for enhancing Antiretroviral Therapy Outcome Prediction for HIV-1
• Mesh Neural Cellular Automata
• On Positional and Structural Node Features for Graph Neural Networks on Non-attributed Graphs
• Towards Representation Identical Privacy-Preserving Graph Neural Network via Split Learning
• Network In Graph Neural Network
• Modeling Ride-Sourcing Matching and Pickup Processes based on Additive Gaussian Process Models
• Dealing with Sparse Rewards Using Graph Neural Networks
• Automated Contrastive Learning Strategy Search for Time Series
• ZIPPER: Exploiting Tile- and Operator-level Parallelism for General and Scalable Graph Neural Network Acceleration
• Evaluating Modules in Graph Contrastive Learning
• A Graph Deep Learning Framework for High-Level Synthesis Design Space Exploration
• Fea2Fea: Exploring Structural Feature Correlations via Graph Neural Networks
• Signed Bipartite Graph Neural Networks
• A Method to Predict Semantic Relations on Artificial Intelligence Papers