Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework

Abstract

Deep Neural Networks (DNNs) have achieved extraordinary performance in various application domains. To support diverse DNN models, efficient implementations of DNN inference on edge-computing platforms, e.g., ASICs, FPGAs, and embedded systems, are extensively investigated. Due to the huge model size and computation amount, model compression is a critical step to deploy DNN models on edge devices. This paper focuses on weight quantization, a hardware-friendly model compression approach that is complementary to weight pruning. Unlike existing methods that use the same quantization scheme for all weights, we propose the first solution that applies different quantization schemes for different rows of the weight matrix. It is motivated by (1) the distribution of the weights in the different rows are not the same; and (2) the potential of achieving better utilization of heterogeneous FPGA hardware resources. To achieve that, we first propose a hardware-friendly quantization scheme named sum-of-power-of-2 (SP2) suitable for Gaussian-like weight distribution, in which the multiplication arithmetic can be replaced with logic shifter and adder, thereby enabling highly efficient implementations with the FPGA LUT resources. In contrast, the existing fixed-point quantization is suitable for Uniform-like weight distribution and can be implemented efficiently by DSP. Then to fully explore the resources, we propose an FPGA-centric mixed scheme quantization (MSQ) with an ensemble of the proposed SP2 and the fixed-point schemes. Combining the two schemes can maintain, or even increase accuracy due to better matching with weight distributions.

Accepted by High-Performance Computer Architecture (HPCA'2021)

References in corpus (17)

• A Survey on Deep Learning in Medical Image Analysis
• YOLOv3: An Incremental Improvement
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Weight Uncertainty in Neural Networks
• Rethinking the Value of Network Pruning
• Ternary Weight Networks
• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• The Microsoft 2017 Conversational Speech Recognition System
• Convolutional Neural Networks using Logarithmic Data Representation
• SlimYOLOv3: Narrower, Faster and Better for Real-Time UAV Applications
• VIBNN: Hardware Acceleration of Bayesian Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Effective Quantization Methods for Recurrent Neural Networks
• A Stochastic-Computing based Deep Learning Framework using Adiabatic Quantum-Flux-Parametron SuperconductingTechnology
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs