WRPN: Wide Reduced-Precision Networks

Abstract

For computer vision applications, prior works have shown the efficacy of reducing numeric precision of model parameters (network weights) in deep neural networks. Activation maps, however, occupy a large memory footprint during both the training and inference step when using mini-batches of inputs. One way to reduce this large memory footprint is to reduce the precision of activations. However, past works have shown that reducing the precision of activations hurts model accuracy. We study schemes to train networks from scratch using reduced-precision activations without hurting accuracy. We reduce the precision of activation maps (along with model parameters) and increase the number of filter maps in a layer, and find that this scheme matches or surpasses the accuracy of the baseline full-precision network. As a result, one can significantly improve the execution efficiency (e.g. reduce dynamic memory footprint, memory bandwidth and computational energy) and speed up the training and inference process with appropriate hardware support. We call our scheme WRPN - wide reduced-precision networks. We report results and show that WRPN scheme is better than previously reported accuracies on ILSVRC-12 dataset while being computationally less expensive compared to previously reported reduced-precision networks.

References in corpus (14)

• Deep Residual Learning for Image Recognition
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Wide Residual Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Going Deeper with Convolutions
• Deep Learning with Limited Numerical Precision
• Ternary Weight Networks
• Trained Ternary Quantization
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Convolutional Neural Networks using Logarithmic Data Representation
• Neural Networks with Few Multiplications
• Resiliency of Deep Neural Networks under Quantization
• Low-Precision Batch-Normalized Activations

Cited by in corpus (64)

• Mixed Precision Training
• Binary Neural Networks: A Survey
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Model compression via distillation and quantization
• A Survey on Methods and Theories of Quantized Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications
• MeliusNet: Can Binary Neural Networks Achieve MobileNet-level Accuracy?
• And the Bit Goes Down: Revisiting the Quantization of Neural Networks
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Quantifying power use in silicon photonic neural networks
• Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
• QKD: Quantization-aware Knowledge Distillation
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• EasyQuant: Post-training Quantization via Scale Optimization
• NICE: Noise Injection and Clamping Estimation for Neural Network Quantization
• Towards Efficient Training for Neural Network Quantization
• Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence
• DSConv: Efficient Convolution Operator
• FINN-R: An End-to-End Deep-Learning Framework for Fast Exploration of Quantized Neural Networks
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• Joint Neural Architecture Search and Quantization
• Differentiable Model Compression via Pseudo Quantization Noise
• HCM: Hardware-Aware Complexity Metric for Neural Network Architectures
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Relaxed Quantization for Discretized Neural Networks
• Iteratively Training Look-Up Tables for Network Quantization
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• Efficient Bitwidth Search for Practical Mixed Precision Neural Network
• CPT: Efficient Deep Neural Network Training via Cyclic Precision
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Full deep neural network training on a pruned weight budget
• Precision Highway for Ultra Low-Precision Quantization
• HEMP: High-order Entropy Minimization for neural network comPression
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• CodeX: Bit-Flexible Encoding for Streaming-based FPGA Acceleration of DNNs
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Post-training Quantization with Multiple Points: Mixed Precision without Mixed Precision
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation
• Distributed Low Precision Training Without Mixed Precision
• Distribution-sensitive Information Retention for Accurate Binary Neural Network
• Neural Network Training on In-memory-computing Hardware with Radix-4 Gradients
• RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• Generative Design of Hardware-aware DNNs
• Fast Adjustable Threshold For Uniform Neural Network Quantization (Winning solution of LPIRC-II)
• Searching for Accurate Binary Neural Architectures
• AskewSGD : An Annealed interval-constrained Optimisation method to train Quantized Neural Networks
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference
• SoFAr: Shortcut-based Fractal Architectures for Binary Convolutional Neural Networks
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks
• BitSplit-Net: Multi-bit Deep Neural Network with Bitwise Activation Function
• Smoothed Differential Privacy
• Light Multi-segment Activation for Model Compression