Binary Neural Networks: A Survey

Abstract

The binary neural network, largely saving the storage and computation, serves as a promising technique for deploying deep models on resource-limited devices. However, the binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network. To address these issues, a variety of algorithms have been proposed, and achieved satisfying progress in recent years. In this paper, we present a comprehensive survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error. We also investigate other practical aspects of binary neural networks such as the hardware-friendly design and the training tricks. Then, we give the evaluation and discussions on different tasks, including image classification, object detection and semantic segmentation. Finally, the challenges that may be faced in future research are prospected.

References in corpus (34)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• YOLOv3: An Incremental Improvement
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Deep Learning with Limited Numerical Precision
• FINN: A Framework for Fast, Scalable Binarized Neural Network Inference
• Compressing Deep Convolutional Networks using Vector Quantization
• Towards Better Exploiting Convolutional Neural Networks for Remote Sensing Scene Classification
• Binarized Neural Networks
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Quantizing deep convolutional networks for efficient inference: A whitepaper
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Deep Adaptive Feature Embedding with Local Sample Distributions for Person Re-identification
• WRPN: Wide Reduced-Precision Networks
• Bitwise Neural Networks
• Loss-aware Binarization of Deep Networks
• XNOR-Net++: Improved Binary Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• ProxQuant: Quantized Neural Networks via Proximal Operators
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Improved training of binary networks for human pose estimation and image recognition
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Training Competitive Binary Neural Networks from Scratch
• Accurate and Compact Convolutional Neural Networks with Trained Binarization
• FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
• Self-Binarizing Networks
• Training Robust Deep Neural Networks via Adversarial Noise Propagation
• PDA: Progressive Data Augmentation for General Robustness of Deep Neural Networks
• Scaling Binarized Neural Networks on Reconfigurable Logic

Cited by in corpus (63)

• Structured Pruning for Deep Convolutional Neural Networks: A survey
• A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification
• Backdoor Attacks and Countermeasures on Deep Learning: A Comprehensive Review
• A comprehensive review of Binary Neural Network
• A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms
• Deep Learning Techniques for In-Crop Weed Identification: A Review
• Rotated Binary Neural Network
• Distributed CNN Inference on Resource-Constrained UAVs for Surveillance Systems: Design and Optimization
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Physical Side-Channel Attacks on Embedded Neural Networks: A Survey
• A Review of Recent Advances of Binary Neural Networks for Edge Computing
• HashVFL: Defending Against Data Reconstruction Attacks in Vertical Federated Learning
• Forward and Backward Information Retention for Accurate Binary Neural Networks
• Binarized Graph Neural Network
• Quantum-limited stochastic optical neural networks operating at a few quanta per activation
• Scalable Rule-Based Representation Learning for Interpretable Classification
• Learning Interpretable Rules for Scalable Data Representation and Classification
• On Quantizing Implicit Neural Representations
• Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Network
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• BiPointNet: Binary Neural Network for Point Clouds
• Diversifying Sample Generation for Accurate Data-Free Quantization
• One-Bit Compressive Sensing: Can We Go Deep and Blind?
• An Integer Programming Approach to Deep Neural Networks with Binary Activation Functions
• A Reconfigurable Linear RF Analog Processor for Realizing Microwave Artificial Neural Network
• Hardware-friendly Deep Learning by Network Quantization and Binarization
• Diverse Sample Generation: Pushing the Limit of Generative Data-free Quantization
• QuPeD: Quantized Personalization via Distillation with Applications to Federated Learning
• A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
• On the Compressive Power of Boolean Threshold Autoencoders
• Popcorn: Paillier Meets Compression For Efficient Oblivious Neural Network Inference
• How to train accurate BNNs for embedded systems?
• 8-bit Optimizers via Block-wise Quantization
• Dither computing: a hybrid deterministic-stochastic computing framework
• BCNN: Binary Complex Neural Network
• QuPeL: Quantized Personalization with Applications to Federated Learning
• Quantum Annealing Formulation for Binary Neural Networks
• Adiabatic Capacitive Neuron: An Energy-Efficient Functional Unit for Artificial Neural Networks
• Variational Inference for Quantum HyperNetworks
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Attention Mechanism with Energy-Friendly Operations
• Implementation of a Binary Neural Network on a Passive Array of Magnetic Tunnel Junctions
• SiMaN: Sign-to-Magnitude Network Binarization
• Federated Learning via Plurality Vote
• LB-CNN: An Open Source Framework for Fast Training of Light Binary Convolutional Neural Networks using Chainer and Cupy
• Enabling Binary Neural Network Training on the Edge
• Detecting soccer balls with reduced neural networks: a comparison of multiple architectures under constrained hardware scenarios
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Binarized Weight Error Networks With a Transition Regularization Term
• Hardware Architecture of Embedded Inference Accelerator and Analysis of Algorithms for Depthwise and Large-Kernel Convolutions
• Exact Backpropagation in Binary Weighted Networks with Group Weight Transformations
• On the Acceleration of Deep Neural Network Inference using Quantized Compressed Sensing
• VC dimension of partially quantized neural networks in the overparametrized regime
• Binary Neural Network for Speaker Verification
• Optimization of XNOR Convolution for Binary Convolutional Neural Networks on GPU
• Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks
• Demystifying and Generalizing BinaryConnect
• Compressed Object Detection
• Multi-Pretext Attention Network for Few-shot Learning with Self-supervision
• Universal Swarm Computing by Nanorobots
• Training Multi-Layer Binary Neural Networks With Local Binary Error Signals
• Renewal model for dependent binary sequences
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression