Low-Precision Batch-Normalized Activations
arXiv:1702.08231
Abstract
Artificial neural networks can be trained with relatively low-precision floating-point and fixed-point arithmetic, using between one and 16 bits. Previous works have focused on relatively wide-but-shallow, feed-forward networks. We introduce a quantization scheme that is compatible with training very deep neural networks. Quantizing the network activations in the middle of each batch-normalization module can greatly reduce the amount of memory and computational power needed, with little loss in accuracy.
16 pages
References in corpus (9)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Wide Residual Networks
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Deep Learning with Limited Numerical Precision
- Ternary Weight Networks
- Convolutional Neural Networks using Logarithmic Data Representation
- Deep neural networks are robust to weight binarization and other non-linear distortions
- Recurrent Batch Normalization
Cited by in corpus (7)
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- WRPN: Wide Reduced-Precision Networks
- Normalization Techniques in Training DNNs: Methodology, Analysis and Application
- Pruning and Quantization for Deep Neural Network Acceleration: A Survey
- Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
- FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
- Enabling Binary Neural Network Training on the Edge