Event-Enhanced Multi-Modal Spiking Neural Network for Dynamic Obstacle Avoidance

Abstract

Autonomous obstacle avoidance is of vital importance for an intelligent agent such as a mobile robot to navigate in its environment. Existing state-of-the-art methods train a spiking neural network (SNN) with deep reinforcement learning (DRL) to achieve energy-efficient and fast inference speed in complex/unknown scenes. These methods typically assume that the environment is static while the obstacles in real-world scenes are often dynamic. The movement of obstacles increases the complexity of the environment and poses a great challenge to the existing methods. In this work, we approach robust dynamic obstacle avoidance twofold. First, we introduce the neuromorphic vision sensor (i.e., event camera) to provide motion cues complementary to the traditional Laser depth data for handling dynamic obstacles. Second, we develop an DRL-based event-enhanced multimodal spiking actor network (EEM-SAN) that extracts information from motion events data via unsupervised representation learning and fuses Laser and event camera data with learnable thresholding. Experiments demonstrate that our EEM-SAN outperforms state-of-the-art obstacle avoidance methods by a significant margin, especially for dynamic obstacle avoidance.

arXiv commentIn Proceedings of the 31st ACM International Conference on Multimedia (ACM MM 2023)

References in corpus (11)

• Spatio-Temporal Backpropagation for Training High-performance Spiking Neural Networks
• Supervised Learning in Multilayer Spiking Neural Networks
• DIET-SNN: Direct Input Encoding With Leakage and Threshold Optimization in Deep Spiking Neural Networks
• GLIF: A Unified Gated Leaky Integrate-and-Fire Neuron for Spiking Neural Networks
• Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control
• Biologically Inspired Dynamic Thresholds for Spiking Neural Networks
• In the Blink of an Eye: Event-based Emotion Recognition
• Finite Meta-Dynamic Neurons in Spiking Neural Networks for Spatio-temporal Learning
• Learning to Navigate in a VUCA Environment: Hierarchical Multi-expert Approach
• Exploration of Reinforcement Learning for Event Camera using Car-like Robots
• Fully Spiking Variational Autoencoder