FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions

Abstract

Differentiable Neural Architecture Search (DNAS) has demonstrated great success in designing state-of-the-art, efficient neural networks. However, DARTS-based DNAS's search space is small when compared to other search methods', since all candidate network layers must be explicitly instantiated in memory. To address this bottleneck, we propose a memory and computationally efficient DNAS variant: DMaskingNAS. This algorithm expands the search space by up to $10^{14}\times$ over conventional DNAS, supporting searches over spatial and channel dimensions that are otherwise prohibitively expensive: input resolution and number of filters. We propose a masking mechanism for feature map reuse, so that memory and computational costs stay nearly constant as the search space expands. Furthermore, we employ effective shape propagation to maximize per-FLOP or per-parameter accuracy. The searched FBNetV2s yield state-of-the-art performance when compared with all previous architectures. With up to 421$\times$ less search cost, DMaskingNAS finds models with 0.9% higher accuracy, 15% fewer FLOPs than MobileNetV3-Small; and with similar accuracy but 20% fewer FLOPs than Efficient-B0. Furthermore, our FBNetV2 outperforms MobileNetV3 by 2.6% in accuracy, with equivalent model size. FBNetV2 models are open-sourced at https://github.com/facebookresearch/mobile-vision.

8 pages, 10 figures, accepted to CVPR 2020

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Neural Architecture Search: A Survey
• Efficient Neural Architecture Search via Parameter Sharing
• Large-Scale Evolution of Image Classifiers
• SNAS: Stochastic Neural Architecture Search
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Progressive Neural Architecture Search
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• Single-Path NAS: Device-Aware Efficient ConvNet Design

Cited by in corpus (17)

• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers
• Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• AutoHAS: Efficient Hyperparameter and Architecture Search
• DNA: Differentiable Network-Accelerator Co-Search
• Neural Architecture Design for GPU-Efficient Networks
• Few-shot Neural Architecture Search
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• K-shot NAS: Learnable Weight-Sharing for NAS with K-shot Supernets
• Pareto-Frontier-aware Neural Architecture Generation for Diverse Budgets
• FALCON: Lightweight and Accurate Convolution
• Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators
• Pi-NAS: Improving Neural Architecture Search by Reducing Supernet Training Consistency Shift
• Adjoined Networks: A Training Paradigm with Applications to Network Compression
• BN-NAS: Neural Architecture Search with Batch Normalization
• One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking