Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference

Abstract

To realize the promise of ubiquitous embedded deep network inference, it is essential to seek limits of energy and area efficiency. To this end, low-precision networks offer tremendous promise because both energy and area scale down quadratically with the reduction in precision. Here we demonstrate ResNet-18, -34, -50, -152, Inception-v3, Densenet-161, and VGG-16bn networks on the ImageNet classification benchmark that, at 8-bit precision exceed the accuracy of the full-precision baseline networks after one epoch of finetuning, thereby leveraging the availability of pretrained models. We also demonstrate ResNet-18, -34, -50, -152, Densenet-161, and VGG-16bn 4-bit models that match the accuracy of the full-precision baseline networks -- the highest scores to date. Surprisingly, the weights of the low-precision networks are very close (in cosine similarity) to the weights of the corresponding baseline networks, making training from scratch unnecessary. We find that gradient noise due to quantization during training increases with reduced precision, and seek ways to overcome this noise. The number of iterations required by SGD to achieve a given training error is related to the square of (a) the distance of the initial solution from the final plus (b) the maximum variance of the gradient estimates. Therefore, we (a) reduce solution distance by starting with pretrained fp32 precision baseline networks and fine-tuning, and (b) combat gradient noise introduced by quantization by training longer and reducing learning rates. Sensitivity analysis indicates that these simple techniques, coupled with proper activation function range calibration to take full advantage of the limited precision, are sufficient to discover low-precision networks, if they exist, close to fp32 precision baseline networks. The results herein provide evidence that 4-bits suffice for classification.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Don't Decay the Learning Rate, Increase the Batch Size
• Convolutional Neural Networks using Logarithmic Data Representation
• Training and Inference with Integers in Deep Neural Networks
• Averaging Weights Leads to Wider Optima and Better Generalization
• Qualitatively characterizing neural network optimization problems
• WRPN: Wide Reduced-Precision Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• High-Accuracy Low-Precision Training
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights

Cited by in corpus (23)

• Accurate deep neural network inference using computational phase-change memory
• Learned Step Size Quantization
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• Pareto-Optimal Quantized ResNet Is Mostly 4-bit
• Towards Unified INT8 Training for Convolutional Neural Network
• Fighting Quantization Bias With Bias
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• Sparse Weight Activation Training
• MWQ: Multiscale Wavelet Quantized Neural Networks
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions
• Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks
• Dynamic Sparse Graph for Efficient Deep Learning
• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks
• Integer-Only Neural Network Quantization Scheme Based on Shift-Batch-Normalization
• Zero-shot Adversarial Quantization
• HPTQ: Hardware-Friendly Post Training Quantization