Rounding Methods for Neural Networks with Low Resolution Synaptic Weights

Abstract

Neural network algorithms simulated on standard computing platforms typically make use of high resolution weights, with floating-point notation. However, for dedicated hardware implementations of such algorithms, fixed-point synaptic weights with low resolution are preferable. The basic approach of reducing the resolution of the weights in these algorithms by standard rounding methods incurs drastic losses in performance. To reduce the resolution further, in the extreme case even to binary weights, more advanced techniques are necessary. To this end, we propose two methods for mapping neural network algorithms with high resolution weights to corresponding algorithms that work with low resolution weights and demonstrate that their performance is substantially better than standard rounding. We further use these methods to investigate the performance of three common neural network algorithms under fixed memory size of the weight matrix with different weight resolutions. We show that dedicated hardware systems, whose technology dictates very low weight resolutions (be they electronic or biological) could in principle implement the algorithms we study.

Cited by in corpus (15)

• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps
• Neuromorphic Deep Learning Machines
• A Survey on Methods and Theories of Quantized Neural Networks
• Adaptive Extreme Edge Computing for Wearable Devices
• Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines
• A neuromorphic systems approach to in-memory computing with non-ideal memristive devices: From mitigation to exploitation
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Prominent characteristics of recurrent neuronal networks are robust against low synaptic weight resolution
• Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses
• Mixed-precision training of deep neural networks using computational memory
• Neural Network Design for Energy-Autonomous AI Applications using Temporal Encoding
• Rethinking Numerical Representations for Deep Neural Networks
• Learning compositional functions via multiplicative weight updates
• Neural and Synaptic Array Transceiver: A Brain-Inspired Computing Framework for Embedded Learning