Graph-based Spatial-temporal Feature Learning for Neuromorphic Vision Sensing

Abstract

Neuromorphic vision sensing (NVS)\ devices represent visual information as sequences of asynchronous discrete events (a.k.a., "spikes") in response to changes in scene reflectance. Unlike conventional active pixel sensing (APS), NVS allows for significantly higher event sampling rates at substantially increased energy efficiency and robustness to illumination changes. However, feature representation for NVS is far behind its APS-based counterparts, resulting in lower performance in high-level computer vision tasks. To fully utilize its sparse and asynchronous nature, we propose a compact graph representation for NVS, which allows for end-to-end learning with graph convolution neural networks. We couple this with a novel end-to-end feature learning framework that accommodates both appearance-based and motion-based tasks. The core of our framework comprises a spatial feature learning module, which utilizes residual-graph convolutional neural networks (RG-CNN), for end-to-end learning of appearance-based features directly from graphs. We extend this with our proposed Graph2Grid block and temporal feature learning module for efficiently modelling temporal dependencies over multiple graphs and a long temporal extent. We show how our framework can be configured for object classification, action recognition and action similarity labeling. Importantly, our approach preserves the spatial and temporal coherence of spike events, while requiring less computation and memory. The experimental validation shows that our proposed framework outperforms all recent methods on standard datasets. Finally, to address the absence of large real-world NVS datasets for complex recognition tasks, we introduce, evaluate and make available the American Sign Language letters (ASL-DVS), as well as human action dataset (UCF101-DVS, HMDB51-DVS and ASLAN-DVS).

16 pages, 5 figures. This work is a journal extension of our ICCV'19 paper arXiv:1908.06648

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Geometric deep learning: going beyond Euclidean data
• Spectral Networks and Locally Connected Networks on Graphs
• Discrete Signal Processing on Graphs
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• EV-FlowNet: Self-Supervised Optical Flow Estimation for Event-based Cameras
• HFirst: A Temporal Approach to Object Recognition
• Event-Driven Contrastive Divergence for Spiking Neuromorphic Systems
• Phased LSTM: Accelerating Recurrent Network Training for Long or Event-based Sequences
• Learning shape correspondence with anisotropic convolutional neural networks
• Local Spectral Graph Convolution for Point Set Feature Learning
• Spatiotemporal Filtering for Event-Based Action Recognition
• Attention Mechanisms for Object Recognition with Event-Based Cameras

Cited by in corpus (13)

• Human Action Recognition from Various Data Modalities: A Review
• Moving Object Detection for Event-based vision using Graph Spectral Clustering
• Multi-Modal Interaction Graph Convolutional Network for Temporal Language Localization in Videos
• A Review of Graph Neural Networks and Their Applications in Power Systems
• Event Voxel Set Transformer for Spatiotemporal Representation Learning on Event Streams
• EvGNN: An Event-driven Graph Neural Network Accelerator for Edge Vision
• EvRepSL: Event-Stream Representation via Self-Supervised Learning for Event-Based Vision
• Event Transformer+. A multi-purpose solution for efficient event data processing
• TTPOINT: A Tensorized Point Cloud Network for Lightweight Action Recognition with Event Cameras
• Memory-Efficient Graph Convolutional Networks for Object Classification and Detection with Event Cameras
• Moving Object Detection for Event-based Vision using k-means Clustering
• LIAF-Net: Leaky Integrate and Analog Fire Network for Lightweight and Efficient Spatiotemporal Information Processing
• Fourier-based Action Recognition for Wildlife Behavior Quantification with Event Cameras