Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search

Abstract

Neural architecture search (NAS) automates the design of deep neural networks. One of the main challenges in searching complex and non-continuous architectures is to compare the similarity of networks that the conventional Euclidean metric may fail to capture. Optimal transport (OT) is resilient to such complex structure by considering the minimal cost for transporting a network into another. However, the OT is generally not negative definite which may limit its ability to build the positive-definite kernels required in many kernel-dependent frameworks. Building upon tree-Wasserstein (TW), which is a negative definite variant of OT, we develop a novel discrepancy for neural architectures, and demonstrate it within a Gaussian process surrogate model for the sequential NAS settings. Furthermore, we derive a novel parallel NAS, using quality k-determinantal point process on the GP posterior, to select diverse and high-performing architectures from a discrete set of candidates. Empirically, we demonstrate that our TW-based approaches outperform other baselines in both sequential and parallel NAS.

23 pages, camera ready ICML2021

References in corpus (13)

• Practical Bayesian Optimization of Machine Learning Algorithms
• Neural Architecture Search with Reinforcement Learning
• Scalable Bayesian Optimization Using Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• SNAS: Stochastic Neural Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Evaluating the Search Phase of Neural Architecture Search
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Batched Gaussian Process Bandit Optimization via Determinantal Point Processes
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
• Tree-Wasserstein Barycenter for Large-Scale Multilevel Clustering and Scalable Bayes
• Entropy Partial Transport with Tree Metrics: Theory and Practice

Cited by in corpus (3)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Fisher Task Distance and Its Application in Neural Architecture Search
• Spectrum Gaussian Processes Based On Tunable Basis Functions