Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels

Abstract

Current neural architecture search (NAS) strategies focus only on finding a single, good, architecture. They offer little insight into why a specific network is performing well, or how we should modify the architecture if we want further improvements. We propose a Bayesian optimisation (BO) approach for NAS that combines the Weisfeiler-Lehman graph kernel with a Gaussian process surrogate. Our method optimises the architecture in a highly data-efficient manner: it is capable of capturing the topological structures of the architectures and is scalable to large graphs, thus making the high-dimensional and graph-like search spaces amenable to BO. More importantly, our method affords interpretability by discovering useful network features and their corresponding impact on the network performance. Indeed, we demonstrate empirically that our surrogate model is capable of identifying useful motifs which can guide the generation of new architectures. We finally show that our method outperforms existing NAS approaches to achieve the state of the art on both closed- and open-domain search spaces.

ICLR 2021. 9 pages, 5 figures, 1 table (23 pages, 14 figures and 3 tables including references and appendices)

References in corpus (22)

• Neural Architecture Search with Reinforcement Learning
• Axiomatic Attribution for Deep Networks
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Understanding Black-box Predictions via Influence Functions
• Large-Scale Evolution of Image Classifiers
• Neural Architecture Optimization
• A Survey on Graph Kernels
• Algorithms for Learning Kernels Based on Centered Alignment
• Efficient Architecture Search by Network Transformation
• SNAS: Stochastic Neural Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Random Search and Reproducibility for Neural Architecture Search
• On Valid Optimal Assignment Kernels and Applications to Graph Classification
• The Multiscale Laplacian Graph Kernel
• Graph Kernels: A Survey
• A Simple Baseline Algorithm for Graph Classification
• Combinatorial Bayesian Optimization using the Graph Cartesian Product
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Graph Structure of Neural Networks
• Asynchronous Batch Bayesian Optimisation with Improved Local Penalisation

Cited by in corpus (7)

• Think Global and Act Local: Bayesian Optimisation over High-Dimensional Categorical and Mixed Search Spaces
• Stronger NAS with Weaker Predictors
• Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search
• CATE: Computation-aware Neural Architecture Encoding with Transformers
• Adversarial Attacks on Graph Classification via Bayesian Optimisation
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-cost Proxies