BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures

Abstract

Binary Neural Networks (BNNs) have received significant attention due to their promising efficiency. Currently, most BNN studies directly adopt widely-used CNN architectures, which can be suboptimal for BNNs. This paper proposes a novel Binary ARchitecture Search (BARS) flow to discover superior binary architecture in a large design space. Specifically, we analyze the information bottlenecks that are related to both the topology and layout architecture design choices. And we propose to automatically search for the optimal information flow. To achieve that, we design a two-level (Macro & Micro) search space tailored for BNNs and apply a differentiable neural architecture search (NAS) to explore this search space efficiently. The macro-level search space includes width and depth decisions, which is required for better balancing the model performance and complexity. We also design the micro-level search space to strengthen the information flow for BNN. %A notable challenge of BNN architecture search lies in that binary operations exacerbate the "collapse" problem of differentiable NAS, for which we incorporate various search and derive strategies to stabilize the search process. On CIFAR-10, BARS achieves 1.5% higher accuracy with 2/3 binary operations and 1/10 floating-point operations comparing with existing BNN NAS studies. On ImageNet, with similar resource consumption, BARS-discovered architecture achieves a 6% accuracy gain than hand-crafted binary ResNet-18 architectures and outperforms other binary architectures while fully binarizing the architecture backbone.

References in corpus (10)

• Categorical Reparameterization with Gumbel-Softmax
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Learning Structured Sparsity in Deep Neural Networks
• SNAS: Stochastic Neural Architecture Search
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization
• Training Competitive Binary Neural Networks from Scratch
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Widening and Squeezing: Towards Accurate and Efficient QNNs

Cited by in corpus (1)

• BoolNet: Minimizing The Energy Consumption of Binary Neural Networks