A Novel Genetic Algorithm with Hierarchical Evaluation Strategy for Hyperparameter Optimisation of Graph Neural Networks

Abstract

Graph representation of structured data can facilitate the extraction of stereoscopic features, and it has demonstrated excellent ability when working with deep learning systems, the so-called Graph Neural Networks (GNNs). Choosing a promising architecture for constructing GNNs can be transferred to a hyperparameter optimisation problem, a very challenging task due to the size of the underlying search space and high computational cost for evaluating candidate GNNs. To address this issue, this research presents a novel genetic algorithm with a hierarchical evaluation strategy (HESGA), which combines the full evaluation of GNNs with a fast evaluation approach. By using full evaluation, a GNN is represented by a set of hyperparameter values and trained on a specified dataset, and root mean square error (RMSE) will be used to measure the quality of the GNN represented by the set of hyperparameter values (for regression problems). While in the proposed fast evaluation process, the training will be interrupted at an early stage, the difference of RMSE values between the starting and interrupted epochs will be used as a fast score, which implies the potential of the GNN being considered. To coordinate both types of evaluations, the proposed hierarchical strategy uses the fast evaluation in a lower level for recommending candidates to a higher level, where the full evaluation will act as a final assessor to maintain a group of elite individuals. To validate the effectiveness of HESGA, we apply it to optimise two types of deep graph neural networks. The experimental results on three benchmark datasets demonstrate its advantages compared to Bayesian hyperparameter optimization.

References in corpus (8)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Graph Neural Networks: A Review of Methods and Applications
• Scalable Bayesian Optimization Using Deep Neural Networks
• Hyperparameter Search in Machine Learning
• Order Matters: Sequence to sequence for sets
• Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter Search
• Efficient Hyperparameter Optimization in Deep Learning Using a Variable Length Genetic Algorithm
• ImmuNeCS: Neural Committee Search by an Artificial Immune System

Cited by in corpus (4)

• Trustworthy Graph Neural Networks: Aspects, Methods and Trends
• Automated Machine Learning on Graphs: A Survey
• A Genetic Algorithm with Tree-structured Mutation for Hyperparameter Optimisation of Graph Neural Networks
• Which Hyperparameters to Optimise? An Investigation of Evolutionary Hyperparameter Optimisation in Graph Neural Network For Molecular Property Prediction