End-to-end Learning of Driving Models from Large-scale Video Datasets

Abstract

Robust perception-action models should be learned from training data with diverse visual appearances and realistic behaviors, yet current approaches to deep visuomotor policy learning have been generally limited to in-situ models learned from a single vehicle or a simulation environment. We advocate learning a generic vehicle motion model from large scale crowd-sourced video data, and develop an end-to-end trainable architecture for learning to predict a distribution over future vehicle egomotion from instantaneous monocular camera observations and previous vehicle state. Our model incorporates a novel FCN-LSTM architecture, which can be learned from large-scale crowd-sourced vehicle action data, and leverages available scene segmentation side tasks to improve performance under a privileged learning paradigm.

camera ready for CVPR2017

References in corpus (8)

• End to End Learning for Self-Driving Cars
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Dynamic Filter Networks
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• An Empirical Evaluation of Deep Learning on Highway Driving
• Unsupervised Learning for Physical Interaction through Video Prediction
• Learning a Driving Simulator
• Learning Transferable Policies for Monocular Reactive MAV Control

Cited by in corpus (20)

• Virtual to Real Reinforcement Learning for Autonomous Driving
• End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies
• A Twofold Siamese Network for Real-Time Object Tracking
• Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars
• MODNet: Moving Object Detection Network with Motion and Appearance for Autonomous Driving
• End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception
• IDK Cascades: Fast Deep Learning by Learning not to Overthink
• Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments
• Fast Recurrent Fully Convolutional Networks for Direct Perception in Autonomous Driving
• Gradient-free Policy Architecture Search and Adaptation
• Brain Inspired Cognitive Model with Attention for Self-Driving Cars
• Imitation Learning for End to End Vehicle Longitudinal Control with Forward Camera
• LIDAR-based Driving Path Generation Using Fully Convolutional Neural Networks
• A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
• Automating Vehicles by Deep Reinforcement Learning using Task Separation with Hill Climbing
• Feature Analysis and Selection for Training an End-to-End Autonomous Vehicle Controller Using the Deep Learning Approach
• Towards a Better Match in Siamese Network Based Visual Object Tracker
• nn-dependability-kit: Engineering Neural Networks for Safety-Critical Autonomous Driving Systems
• Learning On-Road Visual Control for Self-Driving Vehicles with Auxiliary Tasks
• End-to-End Race Driving with Deep Reinforcement Learning