Unsupervised Learning of a Hierarchical Spiking Neural Network for Optical Flow Estimation: From Events to Global Motion Perception

Abstract

The combination of spiking neural networks and event-based vision sensors holds the potential of highly efficient and high-bandwidth optical flow estimation. This paper presents the first hierarchical spiking architecture in which motion (direction and speed) selectivity emerges in an unsupervised fashion from the raw stimuli generated with an event-based camera. A novel adaptive neuron model and stable spike-timing-dependent plasticity formulation are at the core of this neural network governing its spike-based processing and learning, respectively. After convergence, the neural architecture exhibits the main properties of biological visual motion systems, namely feature extraction and local and global motion perception. Convolutional layers with input synapses characterized by single and multiple transmission delays are employed for feature and local motion perception, respectively; while global motion selectivity emerges in a final fully-connected layer. The proposed solution is validated using synthetic and real event sequences. Along with this paper, we provide the cuSNN library, a framework that enables GPU-accelerated simulations of large-scale spiking neural networks. Source code and samples are available at https://github.com/tudelft/cuSNN.

arXiv comment14 pages, 14 figures

References in corpus (12)

• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• STDP-based spiking deep convolutional neural networks for object recognition
• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• SLAYER: Spike Layer Error Reassignment in Time
• Event-based Moving Object Detection and Tracking
• A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation
• Vertical Landing for Micro Air Vehicles using Event-Based Optical Flow
• Bioinspired Visual Motion Estimation
• Efficient Computation in Adaptive Artificial Spiking Neural Networks
• Local Histogram Matching for Efficient Optical Flow Computation Applied to Velocity Estimation on Pocket Drones
• ABMOF: A Novel Optical Flow Algorithm for Dynamic Vision Sensors

Cited by in corpus (14)

• Event-based Vision: A Survey
• Event-Based Fusion for Motion Deblurring with Cross-modal Attention
• Learning Dense and Continuous Optical Flow from an Event Camera
• Event Collapse in Contrast Maximization Frameworks
• ES-ImageNet: A Million Event-Stream Classification Dataset for Spiking Neural Networks
• Voltage-Dependent Synaptic Plasticity (VDSP): Unsupervised probabilistic Hebbian plasticity rule based on neurons membrane potential
• Fast Event-based Optical Flow Estimation by Triplet Matching
• An optimised deep spiking neural network architecture without gradients
• Evolved Neuromorphic Control for High Speed Divergence-based Landings of MAVs
• Time-coded Spiking Fourier Transform in Neuromorphic Hardware
• Back to Event Basics: Self-Supervised Learning of Image Reconstruction for Event Cameras via Photometric Constancy
• SpikeMOT: Event-based Multi-Object Tracking with Sparse Motion Features
• Efficient Implementation of a Multi-Layer Gradient-Free Online-Trainable Spiking Neural Network on FPGA
• Distance Surface for Event-Based Optical Flow