Efficient Differentiable Neural Architecture Search with Meta Kernels

Abstract

The searching procedure of neural architecture search (NAS) is notoriously time consuming and cost prohibitive.To make the search space continuous, most existing gradient-based NAS methods relax the categorical choice of a particular operation to a softmax over all possible operations and calculate the weighted sum of multiple features, resulting in a large memory requirement and a huge computation burden. In this work, we propose an efficient and novel search strategy with meta kernels. We directly encode the supernet from the perspective on convolution kernels and "shrink" multiple convolution kernel candidates into a single one before these candidates operate on the input feature. In this way, only a single feature is generated between two intermediate nodes. The memory for storing intermediate features and the resource budget for conducting convolution operations are both reduced remarkably. Despite high efficiency, our search strategy can search in a more fine-grained way than existing works and increases the capacity for representing possible networks. We demonstrate the effectiveness of our search strategy by conducting extensive experiments. Specifically, our method achieves 77.0% top-1 accuracy on ImageNet benchmark dataset with merely 357M FLOPs, outperforming both EfficientNet and MobileNetV3 under the same FLOPs constraints. Compared to models discovered by the start-of-the-art NAS method, our method achieves the same (sometimes even better) performance, while faster by three orders of magnitude.

References in corpus (22)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• YOLOv3: An Incremental Improvement
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Res2Net: A New Multi-scale Backbone Architecture
• Categorical Reparameterization with Gumbel-Softmax
• AutoAugment: Learning Augmentation Policies from Data
• Convolutional neural networks with low-rank regularization
• Deep High-Resolution Representation Learning for Visual Recognition
• MixConv: Mixed Depthwise Convolutional Kernels
• SNAS: Stochastic Neural Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• Flattened Convolutional Neural Networks for Feedforward Acceleration
• DetNAS: Backbone Search for Object Detection
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Exploring Randomly Wired Neural Networks for Image Recognition
• Deep High-Resolution Representation Learning for Human Pose Estimation
• AutoGAN: Neural Architecture Search for Generative Adversarial Networks
• AdversarialNAS: Adversarial Neural Architecture Search for GANs

Cited by in corpus (7)

• 3D-ANAS: 3D Asymmetric Neural Architecture Search for Fast Hyperspectral Image Classification
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Diverse Branch Block: Building a Convolution as an Inception-like Unit
• AdversarialNAS: Adversarial Neural Architecture Search for GANs
• PONAS: Progressive One-shot Neural Architecture Search for Very Efficient Deployment
• FNA++: Fast Network Adaptation via Parameter Remapping and Architecture Search
• Memory-Efficient Hierarchical Neural Architecture Search for Image Restoration