Generic Neural Architecture Search via Regression

Abstract

Most existing neural architecture search (NAS) algorithms are dedicated to and evaluated by the downstream tasks, e.g., image classification in computer vision. However, extensive experiments have shown that, prominent neural architectures, such as ResNet in computer vision and LSTM in natural language processing, are generally good at extracting patterns from the input data and perform well on different downstream tasks. In this paper, we attempt to answer two fundamental questions related to NAS. (1) Is it necessary to use the performance of specific downstream tasks to evaluate and search for good neural architectures? (2) Can we perform NAS effectively and efficiently while being agnostic to the downstream tasks? To answer these questions, we propose a novel and generic NAS framework, termed Generic NAS (GenNAS). GenNAS does not use task-specific labels but instead adopts regression on a set of manually designed synthetic signal bases for architecture evaluation. Such a self-supervised regression task can effectively evaluate the intrinsic power of an architecture to capture and transform the input signal patterns, and allow more sufficient usage of training samples. Extensive experiments across 13 CNN search spaces and one NLP space demonstrate the remarkable efficiency of GenNAS using regression, in terms of both evaluating the neural architectures (quantified by the ranking correlation Spearman's rho between the approximated performances and the downstream task performances) and the convergence speed for training (within a few seconds).

arXiv commentTo appear at NeurIPS 2021, 10 pages + Reference + Appendix

References in corpus (21)

• Neural Architecture Search with Reinforcement Learning
• DARTS: Differentiable Architecture Search
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Large-Scale Evolution of Image Classifiers
• graph2vec: Learning Distributed Representations of Graphs
• PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
• SNAS: Stochastic Neural Architecture Search
• MCUNet: Tiny Deep Learning on IoT Devices
• Understanding and Robustifying Differentiable Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• One-Shot Neural Architecture Search via Self-Evaluated Template Network
• The Evolved Transformer
• Evaluating the Search Phase of Neural Architecture Search
• SkyNet: a Hardware-Efficient Method for Object Detection and Tracking on Embedded Systems
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• Neural Architecture Search without Training
• FEAR: A Simple Lightweight Method to Rank Architectures