Probabilistic Dual Network Architecture Search on Graphs

Abstract

We present the first differentiable Network Architecture Search (NAS) for Graph Neural Networks (GNNs). GNNs show promising performance on a wide range of tasks, but require a large amount of architecture engineering. First, graphs are inherently a non-Euclidean and sophisticated data structure, leading to poor adaptivity of GNN architectures across different datasets. Second, a typical graph block contains numerous different components, such as aggregation and attention, generating a large combinatorial search space. To counter these problems, we propose a Probabilistic Dual Network Architecture Search (PDNAS) framework for GNNs. PDNAS not only optimises the operations within a single graph block (micro-architecture), but also considers how these blocks should be connected to each other (macro-architecture). The dual architecture (micro- and marco-architectures) optimisation allows PDNAS to find deeper GNNs on diverse datasets with better performance compared to other graph NAS methods. Moreover, we use a fully gradient-based search approach to update architectural parameters, making it the first differentiable graph NAS method. PDNAS outperforms existing hand-designed GNNs and NAS results, for example, on the PPI dataset, PDNAS beats its best competitors by 1.67 and 0.17 in F1 scores.

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Architecture Search with Reinforcement Learning
• Neural Message Passing for Quantum Chemistry
• Categorical Reparameterization with Gumbel-Softmax
• DARTS: Differentiable Architecture Search
• Fast Graph Representation Learning with PyTorch Geometric
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Representation Learning on Graphs with Jumping Knowledge Networks
• Predicting multicellular function through multi-layer tissue networks
• Unrolled Generative Adversarial Networks
• SNAS: Stochastic Neural Architecture Search
• Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
• Probabilistic Neural Architecture Search
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Auto-GNN: Neural Architecture Search of Graph Neural Networks

Cited by in corpus (5)

• Automated Machine Learning on Graphs: A Survey
• On Representation Knowledge Distillation for Graph Neural Networks
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Learned Low Precision Graph Neural Networks
• Rapid Model Architecture Adaption for Meta-Learning