BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations

Abstract

Binary Neural Networks (BNNs) have been garnering interest thanks to their compute cost reduction and memory savings. However, BNNs suffer from performance degradation mainly due to the gradient mismatch caused by binarizing activations. Previous works tried to address the gradient mismatch problem by reducing the discrepancy between activation functions used at forward pass and its differentiable approximation used at backward pass, which is an indirect measure. In this work, we use the gradient of smoothed loss function to better estimate the gradient mismatch in quantized neural network. Analysis using the gradient mismatch estimator indicates that using higher precision for activation is more effective than modifying the differentiable approximation of activation function. Based on the observation, we propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Structured Evolution with Compact Architectures for Scalable Policy Optimization
• Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
• Matrix and tensor decompositions for training binary neural networks

Cited by in corpus (11)

• BinaryBERT: Pushing the Limit of BERT Quantization
• Self-Distribution Binary Neural Networks
• Learning Architectures for Binary Networks
• Generative Zero-shot Network Quantization
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
• PR-CIM: a Variation-Aware Binary-Neural-Network Framework for Process-Resilient Computation-in-memory
• BNAS v2: Learning Architectures for Binary Networks with Empirical Improvements
• VC dimension of partially quantized neural networks in the overparametrized regime
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond