Training and Inference with Integers in Deep Neural Networks

Abstract

Researches on deep neural networks with discrete parameters and their deployment in embedded systems have been active and promising topics. Although previous works have successfully reduced precision in inference, transferring both training and inference processes to low-bitwidth integers has not been demonstrated simultaneously. In this work, we develop a new method termed as "WAGE" to discretize both training and inference, where weights (W), activations (A), gradients (G) and errors (E) among layers are shifted and linearly constrained to low-bitwidth integers. To perform pure discrete dataflow for fixed-point devices, we further replace batch normalization by a constant scaling layer and simplify other components that are arduous for integer implementation. Improved accuracies can be obtained on multiple datasets, which indicates that WAGE somehow acts as a type of regularization. Empirically, we demonstrate the potential to deploy training in hardware systems such as integer-based deep learning accelerators and neuromorphic chips with comparable accuracy and higher energy efficiency, which is crucial to future AI applications in variable scenarios with transfer and continual learning demands.

arXiv comment14 pages, 5 figures, ICLR2018

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Understanding deep learning requires rethinking generalization
• Ternary Weight Networks
• Trained Ternary Quantization
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Convolutional Neural Networks using Logarithmic Data Representation

Cited by in corpus (65)

• Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
• A Survey on Methods and Theories of Quantized Neural Networks
• Scalable Methods for 8-bit Training of Neural Networks
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• Post-training 4-bit quantization of convolution networks for rapid-deployment
• Mixed-precision deep learning based on computational memory
• Bayesian Bits: Unifying Quantization and Pruning
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networks
• Hybrid Tensor Decomposition in Neural Network Compression
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• DSConv: Efficient Convolution Operator
• Dynamically Sacrificing Accuracy for Reduced Computation: Cascaded Inference Based on Softmax Confidence
• Adaptive Neural Network-Based Approximation to Accelerate Eulerian Fluid Simulation
• A Lightweight FPGA-based IDS-ECU Architecture for Automotive CAN
• Compressing 3DCNNs Based on Tensor Train Decomposition
• DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzing
• MARS: Multi-macro Architecture SRAM CIM-Based Accelerator with Co-designed Compressed Neural Networks
• A Lightweight Multi-Attack CAN Intrusion Detection System on Hybrid FPGAs
• Neural-PIM: Efficient Processing-In-Memory with Neural Approximation of Peripherals
• DeepTwist: Learning Model Compression via Occasional Weight Distortion
• Resource-Efficient Neural Networks for Embedded Systems
• Differentiable Model Compression via Pseudo Quantization Noise
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Deep Learning-based Embedded Intrusion Detection System for Automotive CAN
• Relaxed Quantization for Discretized Neural Networks
• Mixed-Precision Embedding Using a Cache
• Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• Quantized Densely Connected U-Nets for Efficient Landmark Localization
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Sparse Weight Activation Training
• Exploring Highly Quantised Neural Networks for Intrusion Detection in Automotive CAN
• Neural Network Design for Energy-Autonomous AI Applications using Temporal Encoding
• Low-Precision Floating-Point Schemes for Neural Network Training
• Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference?
• Latency-Constrained DNN Architecture Learning for Edge Systems using Zerorized Batch Normalization
• DNN+NeuroSim V2.0: An End-to-End Benchmarking Framework for Compute-in-Memory Accelerators for On-chip Training
• TaxoNN: A Light-Weight Accelerator for Deep Neural Network Training
• Training Deep Neural Networks Using Posit Number System
• Improved Lite Audio-Visual Speech Enhancement
• Real-Time Zero-Day Intrusion Detection System for Automotive Controller Area Network on FPGAs
• Differentiable Joint Pruning and Quantization for Hardware Efficiency
• Learning compositional functions via multiplicative weight updates
• Neural gradients are near-lognormal: improved quantized and sparse training
• QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware
• Dynamic Sparse Graph for Efficient Deep Learning
• No Multiplication? No Floating Point? No Problem! Training Networks for Efficient Inference
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• Optimal Quantization for Batch Normalization in Neural Network Deployments and Beyond
• TReCiM: Lower Power and Temperature-Resilient Multibit 2FeFET-1T Compute-in-Memory Design
• Neural Network Training on In-memory-computing Hardware with Radix-4 Gradients
• Training Sparse Neural Networks using Compressed Sensing
• Quantized Adam with Error Feedback
• Efficient and Robust Machine Learning for Real-World Systems
• Group Pruning using a Bounded-Lp norm for Group Gating and Regularization
• Dual Precision Deep Neural Network
• Enabling Binary Neural Network Training on the Edge
• Faster Convolution Inference Through Using Pre-Calculated Lookup Tables
• A Survey on Green Deep Learning
• Bayesian Optimized 1-Bit CNNs
• Iterative Training: Finding Binary Weight Deep Neural Networks with Layer Binarization