Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)

Abstract

A growing body of work underlines striking similarities between biological neural networks and recurrent, binary neural networks. A relatively smaller body of work, however, discusses similarities between learning dynamics employed in deep artificial neural networks and synaptic plasticity in spiking neural networks. The challenge preventing this is largely caused by the discrepancy between the dynamical properties of synaptic plasticity and the requirements for gradient backpropagation. Learning algorithms that approximate gradient backpropagation using locally synthesized gradients can overcome this challenge. Here, we show that synthetic gradients enable the derivation of Deep Continuous Local Learning (DECOLLE) in spiking neural networks. DECOLLE is capable of learning deep spatio-temporal representations from spikes relying solely on local information. Synaptic plasticity rules are derived systematically from user-defined cost functions and neural dynamics by leveraging existing autodifferentiation methods of machine learning frameworks. We benchmark our approach on the MNIST and the event-based neuromorphic DvsGesture dataset, on which DECOLLE performs comparably to the state-of-the-art. DECOLLE networks provide continuously learning machines that are relevant to biology and supportive of event-based, low-power computer vision architectures matching the accuracies of conventional computers on tasks where temporal precision and speed are essential.

Published in Frontiers in Neuroscience - Neuromorphic Engineering

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Striving for Simplicity: The All Convolutional Net
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Convolutional Networks for Fast, Energy-Efficient Neuromorphic Computing
• Neuromorphic electronic circuits for building autonomous cognitive systems
• SuperSpike: Supervised learning in multi-layer spiking neural networks
• SLAYER: Spike Layer Error Reassignment in Time
• Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)
• Neuromorphic Deep Learning Machines
• Surrogate Gradient Learning in Spiking Neural Networks
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures
• Random feedback weights support learning in deep neural networks
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Decoupled Neural Interfaces using Synthetic Gradients
• Is Neuromorphic MNIST neuromorphic? Analyzing the discriminative power of neuromorphic datasets in the time domain
• SMPLR: Deep SMPL reverse for 3D human pose and shape recovery

Cited by in corpus (44)

• Synaptic Plasticity Dynamics for Deep Continuous Local Learning (DECOLLE)
• The Heidelberg spiking datasets for the systematic evaluation of spiking neural networks
• Hardware Implementation of Deep Network Accelerators Towards Healthcare and Biomedical Applications
• Bottom-up and top-down approaches for the design of neuromorphic processing systems: Tradeoffs and synergies between natural and artificial intelligence
• Federated Learning with Spiking Neural Networks
• Learning without feedback: Fixed random learning signals allow for feedforward training of deep neural networks
• Biologically inspired alternatives to backpropagation through time for learning in recurrent neural nets
• Is Neuromorphic MNIST neuromorphic? Analyzing the discriminative power of neuromorphic datasets in the time domain
• Event-based Gesture Recognition with Dynamic Background Suppression using Smartphone Computational Capabilities
• Q-SpiNN: A Framework for Quantizing Spiking Neural Networks
• STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks
• Keys to Accurate Feature Extraction Using Residual Spiking Neural Networks
• Predictive Coding: a Theoretical and Experimental Review
• Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning
• ETLP: Event-based Three-factor Local Plasticity for online learning with neuromorphic hardware
• Embodied Neuromorphic Artificial Intelligence for Robotics: Perspectives, Challenges, and Research Development Stack
• Loss shaping enhances exact gradient learning with Eventprop in spiking neural networks
• RN-Net: Reservoir Nodes-Enabled Neuromorphic Vision Sensing Network
• Comparing SNNs and RNNs on Neuromorphic Vision Datasets: Similarities and Differences
• Event Transformer+. A multi-purpose solution for efficient event data processing
• Spiking Hyperdimensional Network: Neuromorphic Models Integrated with Memory-Inspired Framework
• Gradient-descent hardware-aware training and deployment for mixed-signal Neuromorphic processors
• Encoding Event-Based Data With a Hybrid SNN Guided Variational Auto-encoder in Neuromorphic Hardware
• Exploiting Neuron and Synapse Filter Dynamics in Spatial Temporal Learning of Deep Spiking Neural Network
• Embodied Neuromorphic Vision with Event-Driven Random Backpropagation
• VOWEL: A Local Online Learning Rule for Recurrent Networks of Probabilistic Spiking Winner-Take-All Circuits
• Fourier-based Action Recognition for Wildlife Behavior Quantification with Event Cameras
• Continual Learning with Neuromorphic Computing: Foundations, Methods, and Emerging Applications
• Bayesian Inference on Binary Spiking Networks Leveraging Nanoscale Device Stochasticity
• Deep Spiking Convolutional Neural Network for Single Object Localization Based On Deep Continuous Local Learning
• Spike Talk: Genesis and Neural Coding Scheme Translations
• Benchmarking Spiking Neural Network Learning Methods with Varying Locality
• Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors
• Mantis: Enabling Energy-Efficient Autonomous Mobile Agents with Spiking Neural Networks
• A temporally and spatially local spike-based backpropagation algorithm to enable training in hardware
• Spiking Neural Networks -- Part I: Detecting Spatial Patterns
• Learning to Time-Decode in Spiking Neural Networks Through the Information Bottleneck
• Hyperdimensional Decoding of Spiking Neural Networks
• Exploring neural oscillations during speech perception via surrogate gradient spiking neural networks
• BiSNN: Training Spiking Neural Networks with Binary Weights via Bayesian Learning
• 2022 Roadmap on Neuromorphic Computing and Engineering
• Three factor delay learning rules for spiking neural networks
• A Quarter of a Century of Neuromorphic Architectures on FPGAs -- an Overview
• TESS: A Scalable Temporally and Spatially Local Learning Rule for Spiking Neural Networks