Fixed-point optimization of deep neural networks with adaptive step size retraining

Abstract

Fixed-point optimization of deep neural networks plays an important role in hardware based design and low-power implementations. Many deep neural networks show fairly good performance even with 2- or 3-bit precision when quantized weights are fine-tuned by retraining. We propose an improved fixedpoint optimization algorithm that estimates the quantization step size dynamically during the retraining. In addition, a gradual quantization scheme is also tested, which sequentially applies fixed-point optimizations from high- to low-precision. The experiments are conducted for feed-forward deep neural networks (FFDNNs), convolutional neural networks (CNNs), and recurrent neural networks (RNNs).

arXiv commentThis paper is accepted in ICASSP 2017

References in corpus (3)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Resiliency of Deep Neural Networks under Quantization

Cited by in corpus (2)

• BitNet: Bit-Regularized Deep Neural Networks
• An Overview of Datatype Quantization Techniques for Convolutional Neural Networks