Towards Accurate Binary Convolutional Neural Network

Abstract

We introduce a novel scheme to train binary convolutional neural networks (CNNs) -- CNNs with weights and activations constrained to {-1,+1} at run-time. It has been known that using binary weights and activations drastically reduce memory size and accesses, and can replace arithmetic operations with more efficient bitwise operations, leading to much faster test-time inference and lower power consumption. However, previous works on binarizing CNNs usually result in severe prediction accuracy degradation. In this paper, we address this issue with two major innovations: (1) approximating full-precision weights with the linear combination of multiple binary weight bases; (2) employing multiple binary activations to alleviate information loss. The implementation of the resulting binary CNN, denoted as ABC-Net, is shown to achieve much closer performance to its full-precision counterpart, and even reach the comparable prediction accuracy on ImageNet and forest trail datasets, given adequate binary weight bases and activations.

Cited by in corpus (49)

• A Survey on Methods and Theories of Quantized Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• Bringing AI To Edge: From Deep Learning's Perspective
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• XNOR Neural Engine: a Hardware Accelerator IP for 21.6 fJ/op Binary Neural Network Inference
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Federated Learning for Healthcare Informatics
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Lightweight Pixel Difference Networks for Efficient Visual Representation Learning
• Probabilistic Binary Neural Networks
• Binary Single-dimensional Convolutional Neural Network for Seizure Prediction
• On-device Training: A First Overview on Existing Systems
• Controlling Information Capacity of Binary Neural Network
• Training Competitive Binary Neural Networks from Scratch
• Quantization for Rapid Deployment of Deep Neural Networks
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation
• Learning to Train a Binary Neural Network
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• Sparse Weight Activation Training
• HEMP: High-order Entropy Minimization for neural network comPression
• Accelerating Monte Carlo Bayesian Inference via Approximating Predictive Uncertainty over Simplex
• Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
• Simultaneously Optimizing Weight and Quantizer of Ternary Neural Network using Truncated Gaussian Approximation
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Taming Binarized Neural Networks and Mixed-Integer Programs
• NIPS - Not Even Wrong? A Systematic Review of Empirically Complete Demonstrations of Algorithmic Effectiveness in the Machine Learning and Artificial Intelligence Literature
• RBCN: Rectified Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• Training Progressively Binarizing Deep Networks Using FPGAs
• Approximations in Deep Learning
• Matrix and tensor decompositions for training binary neural networks
• GBCNs: Genetic Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs
• Balanced Binary Neural Networks with Gated Residual
• BinArray: A Scalable Hardware Accelerator for Binary Approximated CNNs
• Composite Binary Decomposition Networks
• Efficient and Robust Machine Learning for Real-World Systems
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks
• Bayesian Optimized 1-Bit CNNs
• From Quantized DNNs to Quantizable DNNs
• Circulant Binary Convolutional Networks: Enhancing the Performance of 1-bit DCNNs with Circulant Back Propagation
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Optimize Deep Convolutional Neural Network with Ternarized Weights and High Accuracy
• Adaptive Binary-Ternary Quantization
• Modulated binary cliquenet
• Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization