Pretraining Neural Architecture Search Controllers with Locality-based Self-Supervised Learning
arXiv:2103.08157
Abstract
Neural architecture search (NAS) has fostered various fields of machine learning. Despite its prominent dedications, many have criticized the intrinsic limitations of high computational cost. We aim to ameliorate this by proposing a pretraining scheme that can be generally applied to controller-based NAS. Our method, locality-based self-supervised classification task, leverages the structural similarity of network architectures to obtain good architecture representations. We incorporate our method into neural architecture optimization (NAO) to analyze the pretrained embeddings and its effectiveness and highlight that adding metric learning loss brings a favorable impact on NAS. Our code is available at \url{https://github.com/Multi-Objective-NAS/self-supervised-nas}.
arXiv commentAccepted to NeurIPS 2020 Workshop: Self-Supervised Learning - Theory and Practice
References in corpus (21)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Neural Architecture Search with Reinforcement Learning
- In Defense of the Triplet Loss for Person Re-Identification
- Spectral Networks and Locally Connected Networks on Graphs
- Efficient Neural Architecture Search via Parameter Sharing
- Neural Architecture Optimization
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- How Powerful are Graph Neural Networks?
- Learning Deep Embeddings with Histogram Loss
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- Hierarchical Representations for Efficient Architecture Search
- The Evolved Transformer
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Semi-Supervised Neural Architecture Search
- Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
- A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS
- PyTorch Metric Learning
- Neural Predictor for Neural Architecture Search
- NASGEM: Neural Architecture Search via Graph Embedding Method