Heterogeneous Bitwidth Binarization in Convolutional Neural Networks

Abstract

Recent work has shown that fast, compact low-bitwidth neural networks can be surprisingly accurate. These networks use homogeneous binarization: all parameters in each layer or (more commonly) the whole model have the same low bitwidth (e.g., 2 bits). However, modern hardware allows efficient designs where each arithmetic instruction can have a custom bitwidth, motivating heterogeneous binarization, where every parameter in the network may have a different bitwidth. In this paper, we show that it is feasible and useful to select bitwidths at the parameter granularity during training. For instance a heterogeneously quantized version of modern networks such as AlexNet and MobileNet, with the right mix of 1-, 2- and 3-bit parameters that average to just 1.4 bits can equal the accuracy of homogeneous 2-bit versions of these networks. Further, we provide analyses to show that the heterogeneously binarized systems yield FPGA- and ASIC-based implementations that are correspondingly more efficient in both circuit area and energy efficiency than their homogeneous counterparts.

arXiv commentNIPS 2018 camera ready update

Cited by in corpus (12)

• Flower: A Friendly Federated Learning Research Framework
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Controlling Information Capacity of Binary Neural Network
• Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Weight Normalization based Quantization for Deep Neural Network Compression
• Automating Generation of Low Precision Deep Learning Operators
• Towards Lossless Binary Convolutional Neural Networks Using Piecewise Approximation
• Mixed-Precision Quantized Neural Network with Progressively Decreasing Bitwidth For Image Classification and Object Detection
• NASB: Neural Architecture Search for Binary Convolutional Neural Networks
• On-device Federated Learning with Flower
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization