Learning Delays in Spiking Neural Networks using Dilated Convolutions with Learnable Spacings
arXiv:2306.17670
Abstract
Spiking Neural Networks (SNNs) are a promising research direction for building power-efficient information processing systems, especially for temporal tasks such as speech recognition. In SNNs, delays refer to the time needed for one spike to travel from one neuron to another. These delays matter because they influence the spike arrival times, and it is well-known that spiking neurons respond more strongly to coincident input spikes. More formally, it has been shown theoretically that plastic delays greatly increase the expressivity in SNNs. Yet, efficient algorithms to learn these delays have been lacking. Here, we propose a new discrete-time algorithm that addresses this issue in deep feedforward SNNs using backpropagation, in an offline manner. To simulate delays between consecutive layers, we use 1D convolutions across time. The kernels contain only a few non-zero weights - one per synapse - whose positions correspond to the delays. These positions are learned together with the weights using the recently proposed Dilated Convolution with Learnable Spacings (DCLS). We evaluated our method on three datasets: the Spiking Heidelberg Dataset (SHD), the Spiking Speech Commands (SSC) and its non-spiking version Google Speech Commands v0.02 (GSC) benchmarks, which require detecting temporal patterns. We used feedforward SNNs with two or three hidden fully connected layers, and vanilla leaky integrate-and-fire neurons. We showed that fixed random delays help and that learning them helps even more. Furthermore, our method outperformed the state-of-the-art in the three datasets without using recurrent connections and with substantially fewer parameters. Our work demonstrates the potential of delay learning in developing accurate and precise models for temporal data processing. Our code is based on PyTorch / SpikingJelly and available at: https://github.com/Thvnvtos/SNN-delays
References in corpus (9)
- Surrogate Gradient Learning in Spiking Neural Networks
- Spikformer: When Spiking Neural Network Meets Transformer
- Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks
- Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks
- Empirical study on the efficiency of Spiking Neural Networks with axonal delays, and algorithm-hardware benchmarking
- Beyond Weights: Deep learning in Spiking Neural Networks with pure synaptic-delay training
- Memory via Temporal Delays in weightless Spiking Neural Network
- Dilated Convolution with Learnable Spacings
- MSAT: Biologically Inspired Multi-Stage Adaptive Threshold for Conversion of Spiking Neural Networks