Self-Binarizing Networks

Abstract

We present a method to train self-binarizing neural networks, that is, networks that evolve their weights and activations during training to become binary. To obtain similar binary networks, existing methods rely on the sign activation function. This function, however, has no gradients for non-zero values, which makes standard backpropagation impossible. To circumvent the difficulty of training a network relying on the sign activation function, these methods alternate between floating-point and binary representations of the network during training, which is sub-optimal and inefficient. We approach the binarization task by training on a unique representation involving a smooth activation function, which is iteratively sharpened during training until it becomes a binary representation equivalent to the sign activation function. Additionally, we introduce a new technique to perform binary batch normalization that simplifies the conventional batch normalization by transforming it into a simple comparison operation. This is unlike existing methods, which are forced to the retain the conventional floating-point-based batch normalization. Our binary networks, apart from displaying advantages of lower memory and computation as compared to conventional floating-point and binary networks, also show higher classification accuracy than existing state-of-the-art methods on multiple benchmark datasets.

9 pages, 5 figures

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Ternary Weight Networks
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• HashNet: Deep Learning to Hash by Continuation
• EffNet: An Efficient Structure for Convolutional Neural Networks

Cited by in corpus (10)

• Binary Neural Networks: A Survey
• Forward and Backward Information Retention for Accurate Binary Neural Networks
• FleXOR: Trainable Fractional Quantization
• Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs
• A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
• Training Progressively Binarizing Deep Networks Using FPGAs
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• BCNN: Binary Complex Neural Network
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework