Accurate and Compact Convolutional Neural Networks with Trained Binarization

Abstract

Although convolutional neural networks (CNNs) are now widely used in various computer vision applications, its huge resource demanding on parameter storage and computation makes the deployment on mobile and embedded devices difficult. Recently, binary convolutional neural networks are explored to help alleviate this issue by quantizing both weights and activations with only 1 single bit. However, there may exist a noticeable accuracy degradation when compared with full-precision models. In this paper, we propose an improved training approach towards compact binary CNNs with higher accuracy. Trainable scaling factors for both weights and activations are introduced to increase the value range. These scaling factors will be trained jointly with other parameters via backpropagation. Besides, a specific training algorithm is developed including tight approximation for derivative of discontinuous binarization function and $L_2$ regularization acting on weight scaling factors. With these improvements, the binary CNN achieves 92.3% accuracy on CIFAR-10 with VGG-Small network. On ImageNet, our method also obtains 46.1% top-1 accuracy with AlexNet and 54.2% with Resnet-18 surpassing previous works.

Accepted as an Oral presentation in British Machine Vision Conference (BMVC) 2019

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Binarized Neural Networks
• Tensorizing Neural Networks
• From Hashing to CNNs: Training BinaryWeight Networks via Hashing
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization

Cited by in corpus (8)

• Binary Neural Networks: A Survey
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Lightweight Pixel Difference Networks for Efficient Visual Representation Learning
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• AdaptCL: Adaptive Continual Learning for Tackling Heterogeneity in Sequential Datasets
• High-Capacity Expert Binary Networks
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization