Multimodal End-to-End Autonomous Driving

Abstract

A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.

The paper has been accepted by IEEE Transactions on Intelligent Transportation Systems 2020

References in corpus (20)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• End to End Learning for Self-Driving Cars
• Fully Convolutional Networks for Semantic Segmentation
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Deep Learning Scaling is Predictable, Empirically
• Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
• Learning a Driving Simulator
• Driving Policy Transfer via Modularity and Abstraction
• Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art
• LocNet: Global localization in 3D point clouds for mobile vehicles
• Complex-YOLO: Real-time 3D Object Detection on Point Clouds
• End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies
• Deep Imitative Models for Flexible Inference, Planning, and Control
• End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception
• On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
• Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability
• Imitation Learning for End to End Vehicle Longitudinal Control with Forward Camera
• End-to-end driving simulation via angle branched network
• Towards Self-Supervised High Level Sensor Fusion
• Improved Semantic Stixels via Multimodal Sensor Fusion

Cited by in corpus (21)

• A Survey of End-to-End Driving: Architectures and Training Methods
• A2D2: Audi Autonomous Driving Dataset
• Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
• Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding
• Multimodal Data Integration for Oncology in the Era of Deep Neural Networks: A Review
• Goal-Guided Transformer-Enabled Reinforcement Learning for Efficient Autonomous Navigation
• End-to-end Autonomous Driving with Semantic Depth Cloud Mapping and Multi-agent
• Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
• Formalizing Multimedia Recommendation through Multimodal Deep Learning
• Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision
• Communication Resources Constrained Hierarchical Federated Learning for End-to-End Autonomous Driving
• NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
• Improving Generalization of Transfer Learning Across Domains Using Spatio-Temporal Features in Autonomous Driving
• Guiding Attention in End-to-End Driving Models
• LUMA: A Benchmark Dataset for Learning from Uncertain and Multimodal Data
• OneAdapt: Fast Configuration Adaptation for Video Analytics Applications via Backpropagation
• FiGKD: Fine-Grained Knowledge Distillation via High-Frequency Detail Transfer
• Driving among Flatmobiles: Bird-Eye-View occupancy grids from a monocular camera for holistic trajectory planning
• Efficient and Robust LiDAR-Based End-to-End Navigation
• Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting
• Yaw-Guided Imitation Learning for Autonomous Driving in Urban Environments