PACT: Parameterized Clipping Activation for Quantized Neural Networks

Abstract

Deep learning algorithms achieve high classification accuracy at the expense of significant computation cost. To address this cost, a number of quantization schemes have been proposed - but most of these techniques focused on quantizing weights, which are relatively smaller in size compared to activations. This paper proposes a novel quantization scheme for activations during training - that enables neural networks to work well with ultra low precision weights and activations without any significant accuracy degradation. This technique, PArameterized Clipping acTivation (PACT), uses an activation clipping parameter $\alpha$ that is optimized during training to find the right quantization scale. PACT allows quantizing activations to arbitrary bit precisions, while achieving much better accuracy relative to published state-of-the-art quantization schemes. We show, for the first time, that both weights and activations can be quantized to 4-bits of precision while still achieving accuracy comparable to full precision networks across a range of popular models and datasets. We also show that exploiting these reduced-precision computational units in hardware can enable a super-linear improvement in inferencing performance due to a significant reduction in the area of accelerator compute engines coupled with the ability to retain the quantized model and activation data in on-chip memories.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Convolutional Sequence to Sequence Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Convolutional Neural Networks using Logarithmic Data Representation
• Bitwise Neural Networks
• Ternary Neural Networks with Fine-Grained Quantization
• Low-Precision Batch-Normalized Activations

Cited by in corpus (200)

• Binary Neural Networks: A Survey
• Learned Step Size Quantization
• MCUNet: Tiny Deep Learning on IoT Devices
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• A Microprocessor implemented in 65nm CMOS with Configurable and Bit-scalable Accelerator for Programmable In-memory Computing
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
• Tiny Machine Learning: Progress and Futures
• Discrimination-aware Network Pruning for Deep Model Compression
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Post-training 4-bit quantization of convolution networks for rapid-deployment
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Exploring the Connection Between Binary and Spiking Neural Networks
• Training with Quantization Noise for Extreme Model Compression
• PP-OCR: A Practical Ultra Lightweight OCR System
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays
• Bayesian Bits: Unifying Quantization and Pruning
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Applications and Techniques for Fast Machine Learning in Science
• On the Accuracy of Analog Neural Network Inference Accelerators
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• Trained Quantization Thresholds for Accurate and Efficient Fixed-Point Inference of Deep Neural Networks
• HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Human Activity Recognition on Microcontrollers with Quantized and Adaptive Deep Neural Networks
• Up or Down? Adaptive Rounding for Post-Training Quantization
• MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
• Constructing Energy-efficient Mixed-precision Neural Networks through Principal Component Analysis for Edge Intelligence
• QKD: Quantization-aware Knowledge Distillation
• Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• EasyQuant: Post-training Quantization via Scale Optimization
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Robust Quantization: One Model to Rule Them All
• Searching for Low-Bit Weights in Quantized Neural Networks
• A Learning Framework for n-bit Quantized Neural Networks toward FPGAs
• Towards Efficient Training for Neural Network Quantization
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations
• Memory-Driven Mixed Low Precision Quantization For Enabling Deep Network Inference On Microcontrollers
• Learning Sparse Low-Precision Neural Networks With Learnable Regularization
• Feature Map Transform Coding for Energy-Efficient CNN Inference
• Pareto-Optimal Quantized ResNet Is Mostly 4-bit
• MARS: Multi-macro Architecture SRAM CIM-Based Accelerator with Co-designed Compressed Neural Networks
• Quantune: Post-training Quantization of Convolutional Neural Networks using Extreme Gradient Boosting for Fast Deployment
• CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework
• Computer Vision Model Compression Techniques for Embedded Systems: A Survey
• ABCNet v2: Adaptive Bezier-Curve Network for Real-time End-to-end Text Spotting
• Technical Report: NEMO DNN Quantization for Deployment Model
• Fighting Quantization Bias With Bias
• Towards Unified INT8 Training for Convolutional Neural Network
• Adaptive Low-Precision Training for Embeddings in Click-Through Rate Prediction
• ZeroQ: A Novel Zero Shot Quantization Framework
• Fixed-point Quantization of Convolutional Neural Networks for Quantized Inference on Embedded Platforms
• HAWQV3: Dyadic Neural Network Quantization
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Design Automation for Efficient Deep Learning Computing
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores
• Hardware-Centric AutoML for Mixed-Precision Quantization
• Switchable Precision Neural Networks
• Automatic Mixed-Precision Quantization Search of BERT
• I-BERT: Integer-only BERT Quantization
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• Precision Gating: Improving Neural Network Efficiency with Dynamic Dual-Precision Activations
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• Ultra-low Power Deep Learning-based Monocular Relative Localization Onboard Nano-quadrotors
• Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodes
• BatchQuant: Quantized-for-all Architecture Search with Robust Quantizer
• Generative Low-bitwidth Data Free Quantization
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks
• Efficient Bitwidth Search for Practical Mixed Precision Neural Network
• DyBit: Dynamic Bit-Precision Numbers for Efficient Quantized Neural Network Inference
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• Blended Coarse Gradient Descent for Full Quantization of Deep Neural Networks
• PAMS: Quantized Super-Resolution via Parameterized Max Scale
• Rethinking Differentiable Search for Mixed-Precision Neural Networks
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Automated Design Space Exploration for optimised Deployment of DNN on Arm Cortex-A CPUs
• Edge AI without Compromise: Efficient, Versatile and Accurate Neurocomputing in Resistive Random-Access Memory
• Precision Highway for Ultra Low-Precision Quantization
• Efficient Deep Neural Networks
• CAT: Compression-Aware Training for bandwidth reduction
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training
• Effective Training of Convolutional Neural Networks with Low-bitwidth Weights and Activations
• Any-Precision Deep Neural Networks
• A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• Channel-wise Hessian Aware trace-Weighted Quantization of Neural Networks
• Network Quantization with Element-wise Gradient Scaling
• Sharpness-aware Quantization for Deep Neural Networks
• Analytical Engines With Context-Rich Processing: Towards Efficient Next-Generation Analytics
• INT8 Winograd Acceleration for Conv1D Equipped ASR Models Deployed on Mobile Devices
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• A Heterogeneous Parallel Non-von Neumann Architecture System for Accurate and Efficient Machine Learning Molecular Dynamics
• OMPQ: Orthogonal Mixed Precision Quantization
• Towards Lightweight Speaker Verification via Adaptive Neural Network Quantization
• Bit Error Robustness for Energy-Efficient DNN Accelerators
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• MWQ: Multiscale Wavelet Quantized Neural Networks
• Convolutional Neural Networks Quantization with Attention
• Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Teachers Do More Than Teach: Compressing Image-to-Image Models
• Conditionally Deep Hybrid Neural Networks Across Edge and Cloud
• PP-ShiTu: A Practical Lightweight Image Recognition System
• Energy-Efficient Adaptive Machine Learning on IoT End-Nodes With Class-Dependent Confidence
• Divide and Conquer: Leveraging Intermediate Feature Representations for Quantized Training of Neural Networks
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
• Generative Zero-shot Network Quantization
• Selective Focus: Investigating Semantics Sensitivity in Post-training Quantization for Lane Detection
• Mixed-Precision Quantized Neural Network with Progressively Decreasing Bitwidth For Image Classification and Object Detection
• SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training
• Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers
• Towards Compact Single Image Super-Resolution via Contrastive Self-distillation
• Neural gradients are near-lognormal: improved quantized and sparse training
• End-to-End Learned Image Compression with Quantized Weights and Activations
• BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization
• AUSN: Approximately Uniform Quantization by Adaptively Superimposing Non-uniform Distribution for Deep Neural Networks
• Some Remarks on Replicated Simulated Annealing
• Learnable Companding Quantization for Accurate Low-bit Neural Networks
• MetaMix: Meta-state Precision Searcher for Mixed-precision Activation Quantization
• MSP: An FPGA-Specific Mixed-Scheme, Multi-Precision Deep Neural Network Quantization Framework
• FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation
• Single-Path Mobile AutoML: Efficient ConvNet Design and NAS Hyperparameter Optimization
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• Dynamic Network Quantization for Efficient Video Inference
• Post-Training Quantization for Vision Transformer
• Sequence-level Speaker Change Detection with Difference-based Continuous Integrate-and-fire
• Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes
• Confounding Tradeoffs for Neural Network Quantization
• Fine-grained Data Distribution Alignment for Post-Training Quantization
• One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment
• OverQ: Opportunistic Outlier Quantization for Neural Network Accelerators
• Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization
• An FPGA Accelerated Method for Training Feed-forward Neural Networks Using Alternating Direction Method of Multipliers and LSMR
• Exploring Quantization and Mapping Synergy in Hardware-Aware Deep Neural Network Accelerators
• Always-On 674uW @ 4GOP/s Error Resilient Binary Neural Networks with Aggressive SRAM Voltage Scaling on a 22nm IoT End-Node
• A Very Compact Embedded CNN Processor Design Based on Logarithmic Computing
• Generative Design of Hardware-aware DNNs
• GDRQ: Group-based Distribution Reshaping for Quantization
• AQD: Towards Accurate Fully-Quantized Object Detection
• RTN: Reparameterized Ternary Network
• Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks
• WaveQ: Gradient-Based Deep Quantization of Neural Networks through Sinusoidal Adaptive Regularization
• Post-Training Sparsity-Aware Quantization
• GradFreeBits: Gradient Free Bit Allocation for Dynamic Low Precision Neural Networks
• Improved Techniques for Quantizing Deep Networks with Adaptive Bit-Widths
• Data-free mixed-precision quantization using novel sensitivity metric
• FATNN: Fast and Accurate Ternary Neural Networks
• Pruning Ternary Quantization
• Hardware and software co-optimization for the initialization failure of the ReRAM based cross-bar array
• Arch-Net: Model Distillation for Architecture Agnostic Model Deployment
• End-to-end Learned Image Compression with Fixed Point Weight Quantization
• Complexity-aware Adaptive Training and Inference for Edge-Cloud Distributed AI Systems
• Scale Calibrated Training: Improving Generalization of Deep Networks via Scale-Specific Normalization
• Hessian-aware Quantized Node Embeddings for Recommendation
• Fully Quantized Image Super-Resolution Networks
• Quantized Neural Network Inference with Precision Batching
• ILMPQ : An Intra-Layer Multi-Precision Deep Neural Network Quantization framework for FPGA
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• PTQ-SL: Exploring the Sub-layerwise Post-training Quantization
• Zero-shot Adversarial Quantization
• Training of mixed-signal optical convolutional neural network with reduced quantization level
• Automatic low-bit hybrid quantization of neural networks through meta learning
• XpulpNN: Enabling Energy Efficient and Flexible Inference of Quantized Neural Network on RISC-V based IoT End Nodes
• Filter Pre-Pruning for Improved Fine-tuning of Quantized Deep Neural Networks
• Learning Quantized Neural Nets by Coarse Gradient Method for Non-linear Classification
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale
• A Mixed-Precision RISC-V Processor for Extreme-Edge DNN Inference
• PokeBNN: A Binary Pursuit of Lightweight Accuracy
• One Shot 3D Photography
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer
• QuantNet: Learning to Quantize by Learning within Fully Differentiable Framework
• HPTQ: Hardware-Friendly Post Training Quantization
• Accelerating Neural Network Inference by Overflow Aware Quantization
• Conditional Neural Architecture Search
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• The data center of tomorrow is made up of heterogeneous accelerators
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization
• Full-stack Optimization for Accelerating CNNs with FPGA Validation
• Towards Mixed-Precision Quantization of Neural Networks via Constrained Optimization
• Low-Complexity LSTM Training and Inference with FloatSD8 Weight Representation
• DupNet: Towards Very Tiny Quantized CNN with Improved Accuracy for Face Detection