Binarized Neural Architecture Search

Abstract

Neural architecture search (NAS) can have a significant impact in computer vision by automatically designing optimal neural network architectures for various tasks. A variant, binarized neural architecture search (BNAS), with a search space of binarized convolutions, can produce extremely compressed models. Unfortunately, this area remains largely unexplored. BNAS is more challenging than NAS due to the learning inefficiency caused by optimization requirements and the huge architecture space. To address these issues, we introduce channel sampling and operation space reduction into a differentiable NAS to significantly reduce the cost of searching. This is accomplished through a performance-based strategy used to abandon less potential operations. Two optimization methods for binarized neural networks are used to validate the effectiveness of our BNAS. Extensive experiments demonstrate that the proposed BNAS achieves a performance comparable to NAS on both CIFAR and ImageNet databases. An accuracy of $96.53\%$ vs. $97.22\%$ is achieved on the CIFAR-10 dataset, but with a significantly compressed model, and a $40\%$ faster search than the state-of-the-art PC-DARTS.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• Improved Regularization of Convolutional Neural Networks with Cutout
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• DARTS: Differentiable Architecture Search
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Efficient Architecture Search by Network Transformation
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation
• Training wide residual networks for deployment using a single bit for each weight

Cited by in corpus (3)

• Learning Architectures for Binary Networks
• Rethinking Performance Estimation in Neural Architecture Search
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements