Toward Computation and Memory Efficient Neural Network Acoustic Models with Binary Weights and Activations
arXiv:1706.09453
Abstract
Neural network acoustic models have significantly advanced state of the art speech recognition over the past few years. However, they are usually computationally expensive due to the large number of matrix-vector multiplications and nonlinearity operations. Neural network models also require significant amounts of memory for inference because of the large model size. For these two reasons, it is challenging to deploy neural network based speech recognizers on resource-constrained platforms such as embedded devices. This paper investigates the use of binary weights and activations for computation and memory efficient neural network acoustic models. Compared to real-valued weight matrices, binary weights require much fewer bits for storage, thereby cutting down the memory footprint. Furthermore, with binary weights or activations, the matrix-vector multiplications are turned into addition and subtraction operations, which are computationally much faster and more energy efficient for hardware platforms. In this paper, we study the applications of binary weights and activations for neural network acoustic modeling, reporting encouraging results on the WSJ and AMI corpora.
arXiv comment7 pages, 3 figures
References in corpus (5)
- Distilling the Knowledge in a Neural Network
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- Do Deep Nets Really Need to be Deep?