Contrastive Embeddings for Neural Architectures
arXiv:2102.04208
Abstract
The performance of algorithms for neural architecture search strongly depends on the parametrization of the search space. We use contrastive learning to identify networks across different initializations based on their data Jacobians, and automatically produce the first architecture embeddings independent from the parametrization of the search space. Using our contrastive embeddings, we show that traditional black-box optimization algorithms, without modification, can reach state-of-the-art performance in Neural Architecture Search. As our method provides a unified embedding space, we perform for the first time transfer learning between search spaces. Finally, we show the evolution of embeddings during training, motivating future studies into using embeddings at different training stages to gain a deeper understanding of the networks in a search space.
arXiv commentAdd "This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 860830"
References in corpus (12)
- A Simple Framework for Contrastive Learning of Visual Representations
- Neural Architecture Search with Reinforcement Learning
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- graph2vec: Learning Distributed Representations of Graphs
- A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
- NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Neural Architecture Search without Training
- Sample-Efficient Neural Architecture Search by Learning Action Space
- Towards NNGP-guided Neural Architecture Search
- Self-supervised Representation Learning for Evolutionary Neural Architecture Search