RC-DARTS: Resource Constrained Differentiable Architecture Search

Abstract

Recent advances show that Neural Architectural Search (NAS) method is able to find state-of-the-art image classification deep architectures. In this paper, we consider the one-shot NAS problem for resource constrained applications. This problem is of great interest because it is critical to choose different architectures according to task complexity when the resource is constrained. Previous techniques are either too slow for one-shot learning or does not take the resource constraint into consideration. In this paper, we propose the resource constrained differentiable architecture search (RC-DARTS) method to learn architectures that are significantly smaller and faster while achieving comparable accuracy. Specifically, we propose to formulate the RC-DARTS task as a constrained optimization problem by adding the resource constraint. An iterative projection method is proposed to solve the given constrained optimization problem. We also propose a multi-level search strategy to enable layers at different depths to adaptively learn different types of neural architectures. Through extensive experiments on the Cifar10 and ImageNet datasets, we show that the RC-DARTS method learns lightweight neural architectures which have smaller model size and lower computational complexity while achieving comparable or better performances than the state-of-the-art methods.

Tech report

References in corpus (14)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• How transferable are features in deep neural networks?
• Improved Regularization of Convolutional Neural Networks with Cutout
• Efficient Neural Architecture Search via Parameter Sharing
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Efficient Architecture Search by Network Transformation
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Hierarchical Representations for Efficient Architecture Search
• Progressive Neural Architecture Search
• DeepArchitect: Automatically Designing and Training Deep Architectures
• IGCV$2$: Interleaved Structured Sparse Convolutional Neural Networks
• Practical Block-wise Neural Network Architecture Generation
• DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures

Cited by in corpus (8)

• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• DNA: Differentiable Network-Accelerator Co-Search
• DASS: Differentiable Architecture Search for Sparse neural networks
• Learning Versatile Neural Architectures by Propagating Network Codes
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Effective, Efficient and Robust Neural Architecture Search
• An Efficient Quantitative Approach for Optimizing Convolutional Neural Networks
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning