Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding

Abstract

This study aims to improve the performance and generalization capability of end-to-end autonomous driving with scene understanding leveraging deep learning and multimodal sensor fusion techniques. The designed end-to-end deep neural network takes as input the visual image and associated depth information in an early fusion level and outputs the pixel-wise semantic segmentation as scene understanding and vehicle control commands concurrently. The end-to-end deep learning-based autonomous driving model is tested in high-fidelity simulated urban driving conditions and compared with the benchmark of CoRL2017 and NoCrash. The testing results show that the proposed approach is of better performance and generalization ability, achieving a 100% success rate in static navigation tasks in both training and unobserved situations, as well as better success rates in other tasks than the prior models. A further ablation study shows that the model with the removal of multimodal sensor fusion or scene understanding pales in the new environment because of the false perception. The results verify that the performance of our model is improved by the synergy of multimodal sensor fusion with scene understanding subtask, demonstrating the feasibility and effectiveness of the developed deep neural network with multimodal sensor fusion.

References in corpus (7)

• End to End Learning for Self-Driving Cars
• Multimodal End-to-End Autonomous Driving
• CARLA: An Open Urban Driving Simulator
• Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
• Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability
• SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning
• Learning by Cheating

Cited by in corpus (14)

• End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent
• Human-in-the-Loop Deep Reinforcement Learning with Application to Autonomous Driving
• A Software Architecture for Autonomous Vehicles: Team LRM-B Entry in the First CARLA Autonomous Driving Challenge
• Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
• Large Language Models and 3D Vision for Intelligent Robotic Perception and Autonomy
• Driving Behavior Modeling using Naturalistic Human Driving Data with Inverse Reinforcement Learning
• DeepIPC: Deeply Integrated Perception and Control for an Autonomous Vehicle in Real Environments
• Engineering nonlinear activation functions for all-optical neural networks via quantum interference
• Low-complexity Point Cloud Filtering for LiDAR by PCA-based Dimension Reduction
• Uncertainty-Aware Model-Based Reinforcement Learning with Application to Autonomous Driving
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments
• DeepIPCv2: LiDAR-powered Robust Environmental Perception and Navigational Control for Autonomous Vehicle
• Seq-DeepIPC: Sequential Sensing for End-to-End Control in Legged Robot Navigation