End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception

Abstract

Convolutional Neural Networks (CNN) have been successfully applied to autonomous driving tasks, many in an end-to-end manner. Previous end-to-end steering control methods take an image or an image sequence as the input and directly predict the steering angle with CNN. Although single task learning on steering angles has reported good performances, the steering angle alone is not sufficient for vehicle control. In this work, we propose a multi-task learning framework to predict the steering angle and speed control simultaneously in an end-to-end manner. Since it is nontrivial to predict accurate speed values with only visual inputs, we first propose a network to predict discrete speed commands and steering angles with image sequences. Moreover, we propose a multi-modal multi-task network to predict speed values and steering angles by taking previous feedback speeds and visual recordings as inputs. Experiments are conducted on the public Udacity dataset and a newly collected SAIC dataset. Results show that the proposed model predicts steering angles and speed values accurately. Furthermore, we improve the failure data synthesis methods to solve the problem of error accumulation in real road tests.

arXiv comment6 pages, 5 figures

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• End to End Learning for Self-Driving Cars
• An Empirical Evaluation of Deep Learning on Highway Driving
• Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car
• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
• Deep Steering: Learning End-to-End Driving Model from Spatial and Temporal Visual Cues
• Interpretable Learning for Self-Driving Cars by Visualizing Causal Attention
• VisualBackProp: efficient visualization of CNNs
• End-to-end Learning of Driving Models from Large-scale Video Datasets
• Brain Inspired Cognitive Model with Attention for Self-Driving Cars

Cited by in corpus (17)

• Multimodal End-to-End Autonomous Driving
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• Towards End-to-End Deep Learning for Autonomous Racing: On Data Collection and a Unified Architecture for Steering and Throttle Prediction
• Imitation Learning for End to End Vehicle Longitudinal Control with Forward Camera
• Affordance-based Reinforcement Learning for Urban Driving
• Closing the gap towards end-to-end autonomous vehicle system
• MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving
• Hierarchical Multi-task Deep Neural Network Architecture for End-to-End Driving
• A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle
• Region-Wise Attack: On Efficient Generation of Robust Physical Adversarial Examples
• Vision-based Navigation of Autonomous Vehicle in Roadway Environments with Unexpected Hazards
• Winning the ICCV 2019 Learning to Drive Challenge
• Accurate Trajectory Prediction for Autonomous Vehicles
• Vehicle Trajectory Prediction by Transfer Learning of Semi-Supervised Models
• Incorporating Orientations into End-to-end Driving Model for Steering Control
• Imitation Learning of Hierarchical Driving Model: from Continuous Intention to Continuous Trajectory
• Vision-Guided Forecasting -- Visual Context for Multi-Horizon Time Series Forecasting