Mixed Precision Training of Convolutional Neural Networks using Integer Operations

Abstract

The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half-precision

arXiv commentPublished as a conference paper at ICLR 2018

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Mixed Precision Training
• 8-Bit Approximations for Parallelism in Deep Learning

Cited by in corpus (29)

• Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
• A Neural-Network-Based Model Predictive Control of Three-Phase Inverter With an Output LC Filter
• Scalable Methods for 8-bit Training of Neural Networks
• Automatic heterogeneous quantization of deep neural networks for low-latency inference on the edge for particle detectors
• TensorDash: Exploiting Sparsity to Accelerate Deep Neural Network Training and Inference
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Mixed Precision Training With 8-bit Floating Point
• NITI: Training Integer Neural Networks Using Integer-only Arithmetic
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems
• SPRING: A Sparsity-Aware Reduced-Precision Monolithic 3D CNN Accelerator Architecture for Training and Inference
• Efficient Visual Recognition with Deep Neural Networks: A Survey on Recent Advances and New Directions
• Quantization of Deep Neural Networks for Accumulator-constrained Processors
• Mixed-precision explicit stabilized Runge-Kutta methods for single- and multi-scale differential equations
• MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions
• Low-Precision Reinforcement Learning: Running Soft Actor-Critic in Half Precision
• Training High-Performance and Large-Scale Deep Neural Networks with Full 8-bit Integers
• Sparse Weight Activation Training
• Rethinking Floating Point Overheads for Mixed Precision DNN Accelerators
• HadaNets: Flexible Quantization Strategies for Neural Networks
• Dynamic Runtime Feature Map Pruning
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off
• Training Deep Neural Network in Limited Precision
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks
• WHO 2016 subtyping and automated segmentation of glioma using multi-task deep learning
• Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training
• Neural Parameter Allocation Search
• NEAT: A Framework for Automated Exploration of Floating Point Approximations
• GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent
• Faster Convolution Inference Through Using Pre-Calculated Lookup Tables