Contrastive Self-supervised Neural Architecture Search

Abstract

This paper proposes a novel cell-based neural architecture search algorithm (NAS), which completely alleviates the expensive costs of data labeling inherited from supervised learning. Our algorithm capitalizes on the effectiveness of self-supervised learning for image representations, which is an increasingly crucial topic of computer vision. First, using only a small amount of unlabeled train data under contrastive self-supervised learning allow us to search on a more extensive search space, discovering better neural architectures without surging the computational resources. Second, we entirely relieve the cost for labeled data (by contrastive loss) in the search stage without compromising architectures' final performance in the evaluation phase. Finally, we tackle the inherent discrete search space of the NAS problem by sequential model-based optimization via the tree-parzen estimator (SMBO-TPE), enabling us to reduce the computational expense response surface significantly. An extensive number of experiments empirically show that our search algorithm can achieve state-of-the-art results with better efficiency in data labeling cost, searching time, and accuracy in final validation.

References in corpus (34)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Representation Learning with Contrastive Predictive Coding
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions
• Bootstrap your own latent: A new approach to self-supervised Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Squeeze-and-Excitation Networks
• Unsupervised Representation Learning by Predicting Image Rotations
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Adversarial Feature Learning
• Efficient Neural Architecture Search via Parameter Sharing
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Efficient Architecture Search by Network Transformation
• BCN20000: Dermoscopic Lesions in the Wild
• SNAS: Stochastic Neural Architecture Search
• Neural Architecture Search with Bayesian Optimisation and Optimal Transport
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Hierarchical Representations for Efficient Architecture Search
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Evaluating the Search Phase of Neural Architecture Search
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• DeepArchitect: Automatically Designing and Training Deep Architectures
• Training Neural Networks with Local Error Signals
• Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data
• NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search
• NSGA-Net: Neural Architecture Search using Multi-Objective Genetic Algorithm
• Probabilistic Neural Architecture Search
• Dual Path Networks
• Self-supervised Neural Architecture Search

Cited by in corpus (4)

• Quantum Embedding Search for Quantum Machine Learning
• Biomarker Discovery with Quantum Neural Networks: A Case-study in CTLA4-Activation Pathways
• COVID-19 Pneumonia Severity Prediction using Hybrid Convolution-Attention Neural Architectures
• Self-Supervised Neural Architecture Search for Multimodal Deep Neural Networks