Smooth Exact Gradient Descent Learning in Spiking Neural Networks
arXiv:2309.14523 · doi:10.1103/PhysRevLett.134.027301
Abstract
Gradient descent prevails in artificial neural network training, but seems inept for spiking neural networks as small parameter changes can cause sudden, disruptive (dis-)appearances of spikes. Here, we demonstrate exact gradient descent based on continuously changing spiking dynamics. These are generated by neuron models whose spikes vanish and appear at the end of a trial, where it cannot influence subsequent dynamics. This also enables gradient-based spike addition and removal. We illustrate our scheme with various tasks and setups, including recurrent and deep, initially silent networks.
References in corpus (12)
- SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
- Array Programming with NumPy
- Neural network models and deep learning - a primer for biologists
- SuperSpike: Supervised learning in multi-layer spiking neural networks
- S4NN: temporal backpropagation for spiking neural networks with one spike per neuron
- Dynamical Entropy Production in Spiking Neuron Networks in the Balanced State
- Event-Based Backpropagation can compute Exact Gradients for Spiking Neural Networks
- Stable Irregular Dynamics in Complex Neural Networks
- Designing the Dynamics of Spiking Neural Networks
- Designing Complex Networks
- How Chaotic is the Balanced State?
- Dynamics and computation in mixed networks containing neurons that accelerate towards spiking