Reduced-Precision Strategies for Bounded Memory in Deep Neural Nets

Abstract

This work investigates how using reduced precision data in Convolutional Neural Networks (CNNs) affects network accuracy during classification. More specifically, this study considers networks where each layer may use different precision data. Our key result is the observation that the tolerance of CNNs to reduced precision data not only varies across networks, a well established observation, but also within networks. Tuning precision per layer is appealing as it could enable energy and performance improvements. In this paper we study how error tolerance across layers varies and propose a method for finding a low precision configuration for a network while maintaining high accuracy. A diverse set of CNNs is analyzed showing that compared to a conventional implementation using a 32-bit floating-point representation for all layers, and with less than 1% loss in relative accuracy, the data footprint required by these networks can be reduced by an average of 74% and up to 92%.

Submitted to ICLR 2016, 12 pages, 5 figures

References in corpus (4)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Deep Learning with Limited Numerical Precision
• Neural Networks with Few Multiplications

Cited by in corpus (14)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey
• Deep Convolutional Neural Network Inference with Floating-point Weights and Fixed-point Activations
• SC-DCNN: Highly-Scalable Deep Convolutional Neural Network using Stochastic Computing
• Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing
• Bit-Tactical: Exploiting Ineffectual Computations in Convolutional Neural Networks: Which, Why, and How
• Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
• BitPruning: Learning Bitlengths for Aggressive and Accurate Quantization
• Low-Precision Floating-Point Schemes for Neural Network Training
• A Simple Method to Reduce Off-chip Memory Accesses on Convolutional Neural Networks
• Convolutional Neural Network Quantization using Generalized Gamma Distribution
• DSLR-CNN: Efficient CNN Acceleration using Digit-Serial Left-to-Right Arithmetic
• Digital Neuron: A Hardware Inference Accelerator for Convolutional Deep Neural Networks
• Memory Requirement Reduction of Deep Neural Networks Using Low-bit Quantization of Parameters