Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

Abstract

Existing Neural Architecture Search (NAS) methods either encode neural architectures using discrete encodings that do not scale well, or adopt supervised learning-based methods to jointly learn architecture representations and optimize architecture search on such representations which incurs search bias. Despite the widespread use, architecture representations learned in NAS are still poorly understood. We observe that the structural properties of neural architectures are hard to preserve in the latent space if architecture representation learning and search are coupled, resulting in less effective search performance. In this work, we find empirically that pre-training architecture representations using only neural architectures without their accuracies as labels considerably improve the downstream architecture search efficiency. To explain these observations, we visualize how unsupervised architecture representation learning better encourages neural architectures with similar connections and operators to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and makes the transition of architectures in the latent space relatively smooth, which considerably benefits diverse downstream search strategies.

arXiv commentNeurIPS 2020 camera-ready. Code: https://github.com/MSU-MLSys-Lab/arch2vec

References in corpus (25)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Inductive Representation Learning on Large Graphs
• Neural Architecture Search with Reinforcement Learning
• Conditional Image Generation with PixelCNN Decoders
• Neural Architecture Search: A Survey
• DARTS: Differentiable Architecture Search
• Momentum Contrast for Unsupervised Visual Representation Learning
• Variational Graph Auto-Encoders
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Optimization
• Scalable Bayesian Optimization Using Deep Neural Networks
• Designing Neural Network Architectures using Reinforcement Learning
• A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
• Adversarial Autoencoders
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Grasp2Vec: Learning Object Representations from Self-Supervised Grasping
• Best Practices for Scientific Research on Neural Architecture Search
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Graph Structure of Neural Networks
• Neural Predictor for Neural Architecture Search

Cited by in corpus (7)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Interpreting the Mechanism of Synergism for Drug Combinations Using Attention-Based Hierarchical Graph Pooling
• Self-supervised Representation Learning for Evolutionary Neural Architecture Search
• Neural Architecture Search with Random Labels
• SEEC: Semantic Vector Federation across Edge Computing Environments
• MONCAE: Multi-Objective Neuroevolution of Convolutional Autoencoders
• Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning