CATE: Computation-aware Neural Architecture Encoding with Transformers

Abstract

Recent works (White et al., 2020a; Yan et al., 2020) demonstrate the importance of architecture encodings in Neural Architecture Search (NAS). These encodings encode either structure or computation information of the neural architectures. Compared to structure-aware encodings, computation-aware encodings map architectures with similar accuracies to the same region, which improves the downstream architecture search performance (Zhang et al., 2019; White et al., 2020a). In this work, we introduce a Computation-Aware Transformer-based Encoding method called CATE. Different from existing computation-aware encodings based on fixed transformation (e.g. path encoding), CATE employs a pairwise pre-training scheme to learn computation-aware encodings using Transformers with cross-attention. Such learned encodings contain dense and contextualized computation information of neural architectures. We compare CATE with eleven encodings under three major encoding-dependent NAS subroutines in both small and large search spaces. Our experiments show that CATE is beneficial to the downstream search, especially in the large search space. Moreover, the outside search space experiment demonstrates its superior generalization ability beyond the search space on which it was trained. Our code is available at: https://github.com/MSU-MLSys-Lab/CATE.

arXiv commentICML 2021 camera-ready. Code: https://github.com/MSU-MLSys-Lab/CATE

References in corpus (29)

• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Optimization
• Scalable Bayesian Optimization Using Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• DeepArchitect: Automatically Designing and Training Deep Architectures
• BayesNAS: A Bayesian Approach for Neural Architecture Search
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search
• Semi-Supervised Neural Architecture Search
• Sample-Efficient Neural Architecture Search by Learning Action Space
• Evaluating Efficient Performance Estimators of Neural Architectures
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Graph Structure of Neural Networks
• Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels
• Neural Architecture Generator Optimization
• A Semi-Supervised Assessor of Neural Architectures
• Can weight sharing outperform random architecture search? An investigation with TuNAS
• BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Contrastive Embeddings for Neural Architectures
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning