Deep Learning with Limited Numerical Precision

Abstract

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network's behavior during training. Our results show that deep networks can be trained using only 16-bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.

10 pages, 6 figures, 1 table

References in corpus (3)

• Improving neural networks by preventing co-adaptation of feature detectors
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Image: Scaling up Image Recognition

Cited by in corpus (173)

• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Mixed Precision Training
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Tensorizing Neural Networks
• Fixed Point Quantization of Deep Convolutional Networks
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Binary Neural Networks: A Survey
• Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices
• Group Sparse Regularization for Deep Neural Networks
• Bayesian Compression for Deep Learning
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• Machine Learning for Microcontroller-Class Hardware: A Review
• Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Scalable Methods for 8-bit Training of Neural Networks
• Pervasive AI for IoT applications: A Survey on Resource-efficient Distributed Artificial Intelligence
• WRPN: Wide Reduced-Precision Networks
• Origami: A 803 GOp/s/W Convolutional Network Accelerator
• Bringing AI To Edge: From Deep Learning's Perspective
• Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks
• Survey of Machine Learning Accelerators
• OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization
• Distilled Siamese Networks for Visual Tracking
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
• Mixed-precision deep learning based on computational memory
• PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions
• DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
• Compressing Recurrent Neural Network with Tensor Train
• The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research
• Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• Rethinking floating point for deep learning
• Relay: A New IR for Machine Learning Frameworks
• 8-Bit Approximations for Parallelism in Deep Learning
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Energy-efficient stochastic computing with superparamagnetic tunnel junctions
• What Do Compressed Deep Neural Networks Forget?
• Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
• Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets
• Robust Ultra-wideband Range Error Mitigation with Deep Learning at the Edge
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Fast Algorithms for Convolutional Neural Networks
• Scaling Deep Learning on GPU and Knights Landing clusters
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Compacting Deep Neural Networks for Internet of Things: Methods and Applications
• DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
• Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework
• Understanding and mitigating noise in trained deep neural networks
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Precision Machine Learning
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• SAFELearning: Enable Backdoor Detectability In Federated Learning With Secure Aggregation
• DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• More is Less: A More Complicated Network with Less Inference Complexity
• Zero-shot Knowledge Transfer via Adversarial Belief Matching
• Low Complexity Multiply Accumulate Unit for Weight-Sharing Convolutional Neural Networks
• Compressing Convolutional Neural Networks
• Adaptive Verifiable Coded Computing: Towards Fast, Secure and Private Distributed Machine Learning
• Distributed Learning in Wireless Networks: Recent Progress and Future Challenges
• Overcoming Challenges in Fixed Point Training of Deep Convolutional Networks
• SPRING: A Sparsity-Aware Reduced-Precision Monolithic 3D CNN Accelerator Architecture for Training and Inference
• Training DNNs with Hybrid Block Floating Point
• Communication-Efficient Edge AI: Algorithms and Systems
• Exploration of Low Numeric Precision Deep Learning Inference Using Intel FPGAs
• EnforceSNN: Enabling Resilient and Energy-Efficient Spiking Neural Network Inference considering Approximate DRAMs for Embedded Systems
• AdaptivFloat: A Floating-point based Data Type for Resilient Deep Learning Inference
• CODEBench: A Neural Architecture and Hardware Accelerator Co-Design Framework
• Optimally Scheduling CNN Convolutions for Efficient Memory Access
• Regularizing Activation Distribution for Training Binarized Deep Networks
• Ultra-compact Binary Neural Networks for Human Activity Recognition on RISC-V Processors
• Evaluating Built-in ECC of FPGA on-chip Memories for the Mitigation of Undervolting Faults
• Accuracy-Efficiency Trade-Offs and Accountability in Distributed ML Systems
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• Fast ConvNets Using Group-wise Brain Damage
• Quantization of Deep Neural Networks for Accumulator-constrained Processors
• Compressive Sensing Using Iterative Hard Thresholding with Low Precision Data Representation: Theory and Applications
• RL-DistPrivacy: Privacy-Aware Distributed Deep Inference for low latency IoT systems
• An OpenCL(TM) Deep Learning Accelerator on Arria 10
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?
• Pre-Trained Models: Past, Present and Future
• ThUnderVolt: Enabling Aggressive Voltage Underscaling and Timing Error Resilience for Energy Efficient Deep Neural Network Accelerators
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• The ZipML Framework for Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning
• A Highly Parallel FPGA Implementation of Sparse Neural Network Training
• lpSpikeCon: Enabling Low-Precision Spiking Neural Network Processing for Efficient Unsupervised Continual Learning on Autonomous Agents
• Prominent characteristics of recurrent neuronal networks are robust against low synaptic weight resolution
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• ModelHub: Towards Unified Data and Lifecycle Management for Deep Learning
• Compacting Neural Network Classifiers via Dropout Training
• Low-Precision Batch-Normalized Activations
• FxP-QNet: A Post-Training Quantizer for the Design of Mixed Low-Precision DNNs with Dynamic Fixed-Point Representation
• Efficient Stochastic Inference of Bitwise Deep Neural Networks
• Mixed-precision training of deep neural networks using computational memory
• Full deep neural network training on a pruned weight budget
• Fast Vocabulary Transfer for Language Model Compression
• Exploring Lottery Ticket Hypothesis in Media Recommender Systems
• SymbolNet: Neural Symbolic Regression with Adaptive Dynamic Pruning for Compression
• Depthwise Multiception Convolution for Reducing Network Parameters without Sacrificing Accuracy
• Deep Learning applied to Road Traffic Speed forecasting
• Memristor-based Deep Convolution Neural Network: A Case Study
• Organ Segmentation From Full-size CT Images Using Memory-Efficient FCN
• Training Bit Fully Convolutional Network for Fast Semantic Segmentation
• A Heterogeneous Parallel Non-von Neumann Architecture System for Accurate and Efficient Machine Learning Molecular Dynamics
• Accurate Sampling with Noisy Forces from Approximate Computing
• Deep Probabilistic Programming Languages: A Qualitative Study
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Reminding Forgetful Organic Neuromorphic Device Networks
• Deep Learning Towards Mobile Applications
• Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• An Automotive Case Study on the Limits of Approximation for Object Detection
• Propagating Asymptotic-Estimated Gradients for Low Bitwidth Quantized Neural Networks
• Understanding the Impact of Precision Quantization on the Accuracy and Energy of Neural Networks
• On the efficient representation and execution of deep acoustic models
• Demystifying the MLPerf Benchmark Suite
• Tensor Decomposition for Compressing Recurrent Neural Network
• Computation Error Analysis of Block Floating Point Arithmetic Oriented Convolution Neural Network Accelerator Design
• Energy-efficient Machine Learning in Silicon: A Communications-inspired Approach
• Reducing the Training Time of Neural Networks by Partitioning
• Tricks from Deep Learning
• Efficient Hardware Realization of Convolutional Neural Networks using Intra-Kernel Regular Pruning
• A Simple Method to Reduce Off-chip Memory Accesses on Convolutional Neural Networks
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• Training Deep Neural Network in Limited Precision
• Convolutional Neural Network Quantization using Generalized Gamma Distribution
• Knowledge distillation for optimization of quantized deep neural networks
• CLAASIC: a Cortex-Inspired Hardware Accelerator
• The Challenge of Multi-Operand Adders in CNNs on FPGAs: How not to solve it!
• LOCAL: Low-Complex Mapping Algorithm for Spatial DNN Accelerators
• A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
• Computational Cost Reduction in Learned Transform Classifications
• ApproxDBN: Approximate Computing for Discriminative Deep Belief Networks
• Neural Network Training on In-memory-computing Hardware with Radix-4 Gradients
• Neural Cache: Bit-Serial In-Cache Acceleration of Deep Neural Networks
• A Selective Survey on Versatile Knowledge Distillation Paradigm for Neural Network Models
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Exploring the Imposition of Synaptic Precision Restrictions For Evolutionary Synthesis of Deep Neural Networks
• ENAS4D: Efficient Multi-stage CNN Architecture Search for Dynamic Inference
• Exploration of Numerical Precision in Deep Neural Networks
• Randomization of Approximate Bilinear Computation for Matrix Multiplication
• Balanced Quantization: An Effective and Efficient Approach to Quantized Neural Networks
• Multi-word Tokenization for Sequence Compression
• An energy-based comparative analysis of common approaches to text classification in the Legal domain
• Recent Advances in Convolutional Neural Network Acceleration
• Mitigate Parasitic Resistance in Resistive Crossbar-based Convolutional Neural Networks
• Characters Detection on Namecard with faster RCNN
• Understanding the Energy and Precision Requirements for Online Learning
• Dual Precision Deep Neural Network
• Training of mixed-signal optical convolutional neural network with reduced quantization level
• A Reconfigurable Streaming Deep Convolutional Neural Network Accelerator for Internet of Things
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Beyond Application End-Point Results: Quantifying Statistical Robustness of MCMC Accelerators
• Full-Stack Filters to Build Minimum Viable CNNs
• Reducing the Model Order of Deep Neural Networks Using Information Theory
• NeuroTrainer: An Intelligent Memory Module for Deep Learning Training
• CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks
• Efficient and Robust Machine Learning for Real-World Systems
• A Main/Subsidiary Network Framework for Simplifying Binary Neural Network
• Selective Deep Convolutional Neural Network for Low Cost Distorted Image Classification
• Light Multi-segment Activation for Model Compression
• GaDei: On Scale-up Training As A Service For Deep Learning
• Fixed-point Factorized Networks
• The data center of tomorrow is made up of heterogeneous accelerators
• Training compact deep learning models for video classification using circulant matrices
• A CNN Accelerator on FPGA Using Depthwise Separable Convolution
• Scientific Calculator for Designing Trojan Detectors in Neural Networks
• Phantom: A High-Performance Computational Core for Sparse Convolutional Neural Networks
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Hardware-Software Codesign of Accurate, Multiplier-free Deep Neural Networks
• Structured Bayesian Compression for Deep models in mobile enabled devices for connected healthcare