GXNOR-Net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework

Abstract

There is a pressing need to build an architecture that could subsume these networks under a unified framework that achieves both higher performance and less overhead. To this end, two fundamental issues are yet to be addressed. The first one is how to implement the back propagation when neuronal activations are discrete. The second one is how to remove the full-precision hidden weights in the training phase to break the bottlenecks of memory/computation consumption. To address the first issue, we present a multi-step neuronal activation discretization method and a derivative approximation technique that enable the implementing the back propagation algorithm on discrete DNNs. While for the second issue, we propose a discrete state transition (DST) methodology to constrain the weights in a discrete space without saving the hidden weights. Through this way, we build a unified framework that subsumes the binary or ternary networks as its special cases, and under which a heuristic algorithm is provided at the website https://github.com/AcrossV/Gated-XNOR. More particularly, we find that when both the weights and activations become ternary values, the DNNs can be reduced to sparse binary networks, termed as gated XNOR networks (GXNOR-Nets) since only the event of non-zero weight and non-zero activation enables the control gate to start the XNOR logic operations in the original binary networks. This promises the event-driven hardware design for efficient mobile intelligence. We achieve advanced performance compared with state-of-the-art algorithms. Furthermore, the computational sparsity and the number of states in the discrete space can be flexibly modified to make it suitable for various hardware platforms.

arXiv comment11 pages, 13 figures

References in corpus (11)

• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Training and Operation of an Integrated Neuromorphic Network Based on Metal-Oxide Memristors
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Ternary Weight Networks
• Trained Ternary Quantization
• Deep Learning using Linear Support Vector Machines
• Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
• Gate-tunable Memristive Phenonmena Mediated by Grain Boundaries in Single Layer MoS2
• Neural Networks with Few Multiplications

Cited by in corpus (2)

• A Survey on Methods and Theories of Quantized Neural Networks
• Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?