Convolutional Neural Networks using Logarithmic Data Representation

Abstract

Recent advances in convolutional neural networks have considered model complexity and hardware efficiency to enable deployment onto embedded systems and mobile devices. For example, it is now well-known that the arithmetic operations of deep networks can be encoded down to 8-bit fixed-point without significant deterioration in performance. However, further reduction in precision down to as low as 3-bit fixed-point results in significant losses in performance. In this paper we propose a new data representation that enables state-of-the-art networks to be encoded to 3 bits with negligible loss in classification performance. To perform this, we take advantage of the fact that the weights and activations in a trained network naturally have non-uniform distributions. Using non-uniform, base-2 logarithmic representation to encode weights, communicate activations, and perform dot-products enables networks to 1) achieve higher classification accuracies than fixed-point at the same resolution and 2) eliminate bulky digital multipliers. Finally, we propose an end-to-end training procedure that uses log representation at 5-bits, which achieves higher final test accuracy than linear at 5-bits.

10 pages, 7 figures

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• cuDNN: Efficient Primitives for Deep Learning
• Tensorizing Neural Networks
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Neural Networks with Few Multiplications
• Resiliency of Deep Neural Networks under Quantization

Cited by in corpus (96)

• PACT: Parameterized Clipping Activation for Quantized Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead
• Scalable Methods for 8-bit Training of Neural Networks
• WRPN: Wide Reduced-Precision Networks
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Rethinking floating point for deep learning
• Training Quantized Nets: A Deeper Understanding
• Loss-aware Weight Quantization of Deep Networks
• On the Accuracy of Analog Neural Network Inference Accelerators
• AddNet: Deep Neural Networks Using FPGA-Optimized Multipliers
• Ternary Neural Networks with Fine-Grained Quantization
• Reducing Computational Complexity of Neural Networks in Optical Channel Equalization: From Concepts to Implementation
• Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks
• ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
• Low-complexity Approximate Convolutional Neural Networks
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
• Energy Efficient Learning with Low Resolution Stochastic Domain Wall Synapse Based Deep Neural Networks
• QUOTIENT: Two-Party Secure Neural Network Training and Prediction
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• MARS: Multi-macro Architecture SRAM CIM-Based Accelerator with Co-designed Compressed Neural Networks
• Mixed Low-precision Deep Learning Inference using Dynamic Fixed Point
• Efficient Spiking Neural Networks with Logarithmic Temporal Coding
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• QGAN: Quantized Generative Adversarial Networks
• Resource-Efficient Neural Networks for Embedded Systems
• A Time-to-first-spike Coding and Conversion Aware Training for Energy-Efficient Deep Spiking Neural Network Processor Design
• N3H-Core: Neuron-designed Neural Network Accelerator via FPGA-based Heterogeneous Computing Cores
• LSQ+: Improving low-bit quantization through learnable offsets and better initialization
• Compressive Sensing Using Iterative Hard Thresholding with Low Precision Data Representation: Theory and Applications
• Switchable Precision Neural Networks
• Post-Training Piecewise Linear Quantization for Deep Neural Networks
• ANNETTE: Accurate Neural Network Execution Time Estimation with Stacked Models
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• Low precision logarithmic number systems: Beyond base-2
• VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inference
• Low-Precision Batch-Normalized Activations
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• tuGEMM: Area-Power-Efficient Temporal Unary GEMM Architecture for Low-Precision Edge AI
• Improving Adversarial Robustness in Weight-quantized Neural Networks
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Iterative Low-Rank Approximation for CNN Compression
• Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
• Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization
• Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework
• MWQ: Multiscale Wavelet Quantized Neural Networks
• Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution
• Ternary Residual Networks
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation
• HarDNet: A Low Memory Traffic Network
• Training Deep Neural Networks Using Posit Number System
• A 1Mb mixed-precision quantized encoder for image classification and patch-based compression
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of Quantization on Depthwise Separable Convolutional Networks Through the Eyes of Multi-scale Distributional Dynamics
• BitNet: Bit-Regularized Deep Neural Networks
• MSP: An FPGA-Specific Mixed-Scheme, Multi-Precision Deep Neural Network Quantization Framework
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• Learnable Companding Quantization for Accurate Low-bit Neural Networks
• Training Deep Neural Network in Limited Precision
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Table-Based Neural Units: Fully Quantizing Networks for Multiply-Free Inference
• One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment
• Histogram-Equalized Quantization for logic-gated Residual Neural Networks
• Neural Machine Translation with 4-Bit Precision and Beyond
• Confounding Tradeoffs for Neural Network Quantization
• n-hot: Efficient bit-level sparsity for powers-of-two neural network quantization
• A Layer-wise Adversarial-aware Quantization Optimization for Improving Robustness
• AQD: Towards Accurate Fully-Quantized Object Detection
• A Very Compact Embedded CNN Processor Design Based on Logarithmic Computing
• DNN Feature Map Compression using Learned Representation over GF(2)
• GDRQ: Group-based Distribution Reshaping for Quantization
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• MUSCO: Multi-Stage Compression of neural networks
• Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression
• Efficient and Robust Machine Learning for Real-World Systems
• PBGen: Partial Binarization of Deconvolution-Based Generators for Edge Intelligence
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Pruning Ternary Quantization
• ILMPQ : An Intra-Layer Multi-Precision Deep Neural Network Quantization framework for FPGA
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• Neural Network Training with Approximate Logarithmic Computations
• Deep Neural Network Training without Multiplications
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks
• Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration
• AutoQNN: An End-to-End Framework for Automatically Quantizing Neural Networks
• Fixed-point Factorized Networks
• NeuroMAX: A High Throughput, Multi-Threaded, Log-Based Accelerator for Convolutional Neural Networks
• Phantom: A High-Performance Computational Core for Sparse Convolutional Neural Networks