Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?

Abstract

Binary neural networks (BNN) have been studied extensively since they run dramatically faster at lower memory and power consumption than floating-point networks, thanks to the efficiency of bit operations. However, contemporary BNNs whose weights and activations are both single bits suffer from severe accuracy degradation. To understand why, we investigate the representation ability, speed and bias/variance of BNNs through extensive experiments. We conclude that the error of BNNs is predominantly caused by the intrinsic instability (training time) and non-robustness (train & test time). Inspired by this investigation, we propose the Binary Ensemble Neural Network (BENN) which leverages ensemble methods to improve the performance of BNNs with limited efficiency cost. While ensemble techniques have been broadly believed to be only marginally helpful for strong classifiers such as deep neural networks, our analyses and experiments show that they are naturally a perfect fit to boost BNNs. We find that our BENN, which is faster and much more robust than state-of-the-art binary networks, can even surpass the accuracy of the full-precision floating number network with the same architecture.

References in corpus (26)

• Adam: A Method for Stochastic Optimization
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Deep Residual Learning for Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• Compressing Deep Convolutional Networks using Vector Quantization
• Binarized Neural Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• Fixed Point Quantization of Deep Convolutional Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Towards Accurate Binary Convolutional Neural Network
• Model compression via distillation and quantization
• Bitwise Neural Networks
• Neural Networks with Few Multiplications
• Resiliency of Deep Neural Networks under Quantization
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Recurrent Neural Networks With Limited Numerical Precision
• Quantized Convolutional Neural Networks for Mobile Devices
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA

Cited by in corpus (8)

• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• Learning Architectures for Binary Networks
• Towards Fast and Energy-Efficient Binarized Neural Network Inference on FPGA
• Stein Variational Inference for Discrete Distributions
• LB-CNN: An Open Source Framework for Fast Training of Light Binary Convolutional Neural Networks using Chainer and Cupy
• Scalable Approximate Inference and Some Applications