BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
arXiv:2002.03090
Abstract
Neural networks have demonstrably achieved state-of-the art accuracy using low-bitlength integer quantization, yielding both execution time and energy benefits on existing hardware designs that support short bitlengths. However, the question of finding the minimum bitlength for a desired accuracy remains open. We introduce a training method for minimizing inference bitlength at any granularity while maintaining accuracy. Namely, we propose a regularizer that penalizes large bitlength representations throughout the architecture and show how it can be modified to minimize other quantifiable criteria, such as number of operations or memory footprint. We demonstrate that our method learns thrifty representations while maintaining accuracy. With ImageNet, the method produces an average per layer bitlength of 4.13, 3.76 and 4.36 bits on AlexNet, ResNet18 and MobileNet V2 respectively, remaining within 2.0%, 0.5% and 0.5% of the base TOP-1 accuracy.
References in corpus (14)
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Densely Connected Convolutional Networks
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Aggregated Residual Transformations for Deep Neural Networks
- Learned Step Size Quantization
- WRPN: Wide Reduced-Precision Networks
- Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets
- ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
- Dual Path Networks
- Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks
- Dynamic Stripes: Exploiting the Dynamic Precision Requirements of Activation Values in Neural Networks