Training Multi-Layer Binary Neural Networks With Local Binary Error Signals

Abstract

Binary Neural Networks (BNNs) significantly reduce computational complexity and memory usage in machine and deep learning by representing weights and activations with just one bit. However, most existing training algorithms for BNNs rely on quantization-aware floating-point Stochastic Gradient Descent (SGD), limiting the full exploitation of binary operations to the inference phase only. In this work, we propose, for the first time, a fully binary and gradient-free training algorithm for multi-layer BNNs, eliminating the need for back-propagated floating-point gradients. Specifically, the proposed algorithm relies on local binary error signals and binary weight updates, employing integer-valued hidden weights that serve as a synaptic metaplasticity mechanism, thereby enhancing its neurobiological plausibility. Our proposed solution enables the training of binary multi-layer perceptrons by using exclusively XNOR, Popcount, and increment/decrement operations. Experimental results on multi-class classification benchmarks show test accuracy improvements of up to +35.47% over the only existing fully binary single-layer state-of-the-art solution. Compared to full-precision SGD, our solution improves test accuracy by up to +35.30% under the same total memory demand, while also reducing computational cost by two to three orders of magnitude in terms of the total number of Boolean gates. The proposed algorithm is made available to the scientific community as a public repository.

References in corpus (15)

• Overcoming catastrophic forgetting in neural networks
• Binary Neural Networks: A Survey
• Learning by message-passing in networks of discrete synapses
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• A comprehensive review of Binary Neural Network
• Efficient supervised learning in networks with binary synapses
• Why should we add early exits to neural networks?
• UNIQ: Uniform Noise Injection for Non-Uniform Quantization of Neural Networks
• Shaping the learning landscape in neural networks around wide flat minima
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• The Backpropagation Algorithm Implemented on Spiking Neuromorphic Hardware
• Generalization learning in a perceptron with binary synapses
• Chaos and correlated avalanches in excitatory neural networks with synaptic plasticity
• Deep learning via message passing algorithms based on belief propagation
• Order symmetry breaking and broad distribution of events in spiking neural networks with continuous membrane potential