NITI: Training Integer Neural Networks Using Integer-only Arithmetic

Abstract

While integer arithmetic has been widely adopted for improved performance in deep quantized neural network inference, training remains a task primarily executed using floating point arithmetic. This is because both high dynamic range and numerical accuracy are central to the success of most modern training algorithms. However, due to its potential for computational, storage and energy advantages in hardware accelerators, neural network training methods that can be implemented with low precision integer-only arithmetic remains an active research challenge. In this paper, we present NITI, an efficient deep neural network training framework that stores all parameters and intermediate values as integers, and computes exclusively with integer arithmetic. A pseudo stochastic rounding scheme that eliminates the need for external random number generation is proposed to facilitate conversion from wider intermediate results to low precision storage. Furthermore, a cross-entropy loss backpropagation scheme computed with integer-only arithmetic is proposed. A proof-of-concept open-source software implementation of NITI that utilizes native 8-bit integer operations in modern GPUs to achieve end-to-end training is presented. When compared with an equivalent training setup implemented with floating point storage and arithmetic, NITI achieves negligible accuracy degradation on the MNIST and CIFAR10 datasets using 8-bit integer storage and computation. On ImageNet, 16-bit integers are needed for weight accumulation with an 8-bit datapath. This achieves training results comparable to all-floating-point implementations.

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• Mixed Precision Training
• Trained Ternary Quantization
• Machine Learning at the Wireless Edge: Distributed Stochastic Gradient Descent Over-the-Air
• Training and Inference with Integers in Deep Neural Networks
• Training deep neural networks with low precision multiplications
• Scalable Methods for 8-bit Training of Neural Networks
• High-Dimensional Stochastic Gradient Quantization for Communication-Efficient Edge Learning
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Rethinking floating point for deep learning

Cited by in corpus (3)

• Advances of Machine Learning in Materials Science: Ideas and Techniques
• PRIOT: Pruning-Based Integer-Only Transfer Learning for Embedded Systems
• Scaling Analog Photonic Accelerators for Byte-Size, Integer General Matrix Multiply (GEMM) Kernels