Regularizing Activation Distribution for Training Binarized Deep Networks

Abstract

Binarized Neural Networks (BNNs) can significantly reduce the inference latency and energy consumption in resource-constrained devices due to their pure-logical computation and fewer memory accesses. However, training BNNs is difficult since the activation flow encounters degeneration, saturation, and gradient mismatch problems. Prior work alleviates these issues by increasing activation bits and adding floating-point scaling factors, thereby sacrificing BNN's energy efficiency. In this paper, we propose to use distribution loss to explicitly regularize the activation flow, and develop a framework to systematically formulate the loss. Our experiments show that the distribution loss can consistently improve the accuracy of BNNs without losing their energy benefits. Moreover, equipped with the proposed regularization, BNN training is shown to be robust to the selection of hyper-parameters including optimizer and learning rate.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Binarized Neural Networks
• The Loss Surfaces of Multilayer Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Self-Normalizing Neural Networks
• Towards Accurate Binary Convolutional Neural Network
• Model compression via distillation and quantization
• All you need is a good init
• Loss-aware Binarization of Deep Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling
• All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation
• Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm
• Intelligence Beyond the Edge: Inference on Intermittent Embedded Systems
• LightNN: Filling the Gap between Conventional Deep Neural Networks and Binarized Networks

Cited by in corpus (4)

• Feature Map Transform Coding for Energy-Efficient CNN Inference
• Weight Normalization based Quantization for Deep Neural Network Compression
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model