BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements

Abstract

Backbone architectures of most binary networks are well-known floating point (FP) architectures such as the ResNet family. Questioning that the architectures designed for FP networks might not be the best for binary networks, we propose to search architectures for binary networks (BNAS) by defining a new search space for binary architectures and a novel search objective. Specifically, based on the cell based search method, we define the new search space of binary layer types, design a new cell template, and rediscover the utility of and propose to use the Zeroise layer instead of using it as a placeholder. The novel search objective diversifies early search to learn better performing binary architectures. We show that our method searches architectures with stable training curves despite the quantization error inherent in binary networks. Quantitative analyses demonstrate that our searched architectures outperform the architectures used in state-of-the-art binary networks and outperform or perform on par with state-of-the-art binary networks that employ various techniques other than architectural changes. In addition, we further propose improvements to the training scheme of our searched architectures. With the new training scheme for our searched architectures, we achieve the state-of-the-art performance by binary networks by outperforming all previous methods by non-trivial margins.

arXiv commentarXiv admin note: text overlap with arXiv:2002.06963

References in corpus (22)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Categorical Reparameterization with Gumbel-Softmax
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• DARTS: Differentiable Architecture Search
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Ternary Weight Networks
• Trained Ternary Quantization
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Neural Architecture Optimization
• Random Search and Reproducibility for Neural Architecture Search
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Exploring Randomly Wired Neural Networks for Image Recognition
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Rotated Binary Neural Network
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Joint Neural Architecture Search and Quantization
• MoBiNet: A Mobile Binary Network for Image Classification
• Binarized Neural Architecture Search