Learning Architectures for Binary Networks

Abstract

Backbone architectures of most binary networks are well-known floating point architectures such as the ResNet family. Questioning that the architectures designed for floating point networks would not be the best for binary networks, we propose to search architectures for binary networks (BNAS) by defining a new search space for binary architectures and a novel search objective. Specifically, based on the cell based search method, we define the new search space of binary layer types, design a new cell template, and rediscover the utility of and propose to use the Zeroise layer instead of using it as a placeholder. The novel search objective diversifies early search to learn better performing binary architectures. We show that our proposed method searches architectures with stable training curves despite the quantization error inherent in binary networks. Quantitative analyses demonstrate that our searched architectures outperform the architectures used in state-of-the-art binary networks and outperform or perform on par with state-of-the-art binary networks that employ various techniques other than architectural changes.

arXiv commentThe manuscript was changed to a one-column format along with minor modifications to the content

References in corpus (26)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• DARTS: Differentiable Architecture Search
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Ternary Weight Networks
• Trained Ternary Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Neural Architecture Optimization
• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• SNAS: Stochastic Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• XNOR-Net++: Improved Binary Neural Networks
• Graph HyperNetworks for Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Joint Neural Architecture Search and Quantization
• MoBiNet: A Mobile Binary Network for Image Classification
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• Binarized Neural Architecture Search