WRPN: Wide Reduced-Precision Networks

Abstract

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

References in corpus (4)

• Going Deeper with Convolutions
• Deep Learning with Limited Numerical Precision
• Trained Ternary Quantization
• Low-Precision Batch-Normalized Activations

Cited by in corpus (20)

• QKD: Quantization-aware Knowledge Distillation
• EasyQuant: Post-training Quantization via Scale Optimization
• Towards Efficient Training for Neural Network Quantization
• Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Precision Highway for Ultra Low-Precision Quantization
• HEMP: High-order Entropy Minimization for neural network comPression
• CodeX: Bit-Flexible Encoding for Streaming-based FPGA Acceleration of DNNs
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• Distributed Low Precision Training Without Mixed Precision
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
• Generative Design of Hardware-aware DNNs
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
• BitSplit-Net: Multi-bit Deep Neural Network with Bitwise Activation Function