Learned Low Precision Graph Neural Networks

Abstract

Deep Graph Neural Networks (GNNs) show promising performance on a range of graph tasks, yet at present are costly to run and lack many of the optimisations applied to DNNs. We show, for the first time, how to systematically quantise GNNs with minimal or no loss in performance using Network Architecture Search (NAS). We define the possible quantisation search space of GNNs. The proposed novel NAS mechanism, named Low Precision Graph NAS (LPGNAS), constrains both architecture and quantisation choices to be differentiable. LPGNAS learns the optimal architecture coupled with the best quantisation strategy for different components in the GNN automatically using back-propagation in a single search round. On eight different datasets, solving the task of classifying unseen nodes in a graph, LPGNAS generates quantised models with significant reductions in both model and buffer sizes but with similar accuracy to manually designed networks and other NAS results. In particular, on the Pubmed dataset, LPGNAS shows a better size-accuracy Pareto frontier compared to seven other manual and searched baselines, offering a 2.3 times reduction in model size but a 0.4% increase in accuracy when compared to the best NAS competitor. Finally, from our collected quantisation statistics on a wide range of datasets, we suggest a W4A8 (4-bit weights, 8-bit activations) quantisation strategy might be the bottleneck for naive GNN quantisations.

References in corpus (22)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Architecture Search with Reinforcement Learning
• Neural Message Passing for Quantum Chemistry
• DARTS: Differentiable Architecture Search
• Fast Graph Representation Learning with PyTorch Geometric
• Binarized Neural Networks
• Trained Ternary Quantization
• Pitfalls of Graph Neural Network Evaluation
• Predicting multicellular function through multi-layer tissue networks
• Large-Scale Evolution of Image Classifiers
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking
• Stochastic Training of Graph Convolutional Networks with Variance Reduction
• Design Space for Graph Neural Networks
• GraphNAS: Graph Neural Architecture Search with Reinforcement Learning
• Auto-GNN: Neural Architecture Search of Graph Neural Networks
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Probabilistic Dual Network Architecture Search on Graphs

Cited by in corpus (6)

• Trustworthy Graph Neural Networks: Aspects, Methods and Trends
• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Automated Machine Learning on Graphs: A Survey
• On Representation Knowledge Distillation for Graph Neural Networks
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• A Survey on Green Deep Learning