Neural Architecture Search with Random Labels
arXiv:2101.11834
Abstract
In this paper, we investigate a new variant of neural architecture search (NAS) paradigm -- searching with random labels (RLNAS). The task sounds counter-intuitive for most existing NAS algorithms since random label provides few information on the performance of each candidate architecture. Instead, we propose a novel NAS framework based on ease-of-convergence hypothesis, which requires only random labels during searching. The algorithm involves two steps: first, we train a SuperNet using random labels; second, from the SuperNet we extract the sub-network whose weights change most significantly during the training. Extensive experiments are evaluated on multiple datasets (e.g. NAS-Bench-201 and ImageNet) and multiple search spaces (e.g. DARTS-like and MobileNet-like). Very surprisingly, RLNAS achieves comparable or even better results compared with state-of-the-art NAS methods such as PC-DARTS, Single Path One-Shot, even though the counterparts utilize full ground truth labels for searching. We hope our finding could inspire new understandings on the essential of NAS.
arXiv commentAccepted in CVPR 2021
References in corpus (18)
- Rethinking Atrous Convolution for Semantic Image Segmentation
- A Simple Framework for Contrastive Learning of Visual Representations
- EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
- Neural Architecture Search with Reinforcement Learning
- Understanding deep learning requires rethinking generalization
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Efficient Neural Architecture Search via Parameter Sharing
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Evaluating the Search Phase of Neural Architecture Search
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Neural Architecture Search without Training
- Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?
- What Do Neural Networks Learn When Trained With Random Labels?
- Angle-based Search Space Shrinking for Neural Architecture Search
- Are Labels Necessary for Neural Architecture Search?