Learning Recurrent Binary/Ternary Weights

Abstract

Recurrent neural networks (RNNs) have shown excellent performance in processing sequence data. However, they are both complex and memory intensive due to their recursive nature. These limitations make RNNs difficult to embed on mobile devices requiring real-time processes with limited hardware resources. To address the above issues, we introduce a method that can learn binary and ternary weights during the training phase to facilitate hardware implementations of RNNs. As a result, using this approach replaces all multiply-accumulate operations by simple accumulations, bringing significant benefits to custom hardware in terms of silicon area and power consumption. On the software side, we evaluate the performance (in terms of accuracy) of our method using long short-term memories (LSTMs) on various sequential models including sequence classification and language modeling. We demonstrate that our method achieves competitive results on the aforementioned tasks while using binary/ternary weights during the runtime. On the hardware side, we present custom hardware for accelerating the recurrent computations of LSTMs with binary/ternary weights. Ultimately, we show that LSTMs with binary/ternary weights can achieve up to 12x memory saving and 10x inference speedup compared to the full-precision implementation on an ASIC platform.

arXiv commentPublished as a conference paper at ICLR 2019

References in corpus (18)

• Sequence to Sequence Learning with Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Teaching Machines to Read and Comprehend
• Recurrent Neural Network Regularization
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Binarized Neural Networks
• Visualizing and Understanding Recurrent Networks
• Ternary Weight Networks
• Trained Ternary Quantization
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Convolutional neural networks with low-rank regularization
• Loss-aware Binarization of Deep Networks
• Loss-aware Weight Quantization of Deep Networks
• Alternating Multi-bit Quantization for Recurrent Neural Networks
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks