DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving

Abstract

Today, there are two major paradigms for vision-based autonomous driving systems: mediated perception approaches that parse an entire scene to make a driving decision, and behavior reflex approaches that directly map an input image to a driving action by a regressor. In this paper, we propose a third paradigm: a direct perception approach to estimate the affordance for driving. We propose to map an input image to a small number of key perception indicators that directly relate to the affordance of a road/traffic state for driving. Our representation provides a set of compact yet complete descriptions of the scene to enable a simple controller to drive autonomously. Falling in between the two extremes of mediated perception and behavior reflex, we argue that our direct perception representation provides the right level of abstraction. To demonstrate this, we train a deep Convolutional Neural Network using recording from 12 hours of human driving in a video game and show that our model can work well to drive a car in a very diverse set of virtual environments. We also train a model for car distance estimation on the KITTI dataset. Results show that our direct perception approach can generalize well to real driving images. Source code and data are available on our project website.

References in corpus (1)

• Caffe: Convolutional Architecture for Fast Feature Embedding

Cited by in corpus (71)

• Guiding Deep Learning System Testing using Surprise Adequacy
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• CARLA: An Open Urban Driving Simulator
• Virtual to Real Reinforcement Learning for Autonomous Driving
• Resiliency of Deep Neural Networks under Quantization
• InfoGAIL: Interpretable Imitation Learning from Visual Demonstrations
• GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
• Query-Efficient Imitation Learning for End-to-End Autonomous Driving
• Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation
• Object Detection with Deep Learning: A Review
• Artificial Intelligence and its Role in Near Future
• Intention-Net: Integrating Planning and Deep Learning for Goal-Directed Autonomous Navigation
• A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation
• NeuronInspect: Detecting Backdoors in Neural Networks via Output Explanations
• Hardware Trojan Attacks on Neural Networks
• Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars
• Training Deep Face Recognition Systems with Synthetic Data
• ExtremeWeather: A large-scale climate dataset for semi-supervised detection, localization, and understanding of extreme weather events
• SegFlow: Joint Learning for Video Object Segmentation and Optical Flow
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• VisualBackProp: efficient visualization of CNNs
• Visual Affordance and Function Understanding: A Survey
• DeepBillboard: Systematic Physical-World Testing of Autonomous Driving Systems
• End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception
• Path Integral Networks: End-to-End Differentiable Optimal Control
• Reinforcement Learning and Deep Learning based Lateral Control for Autonomous Driving
• On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
• End-to-end Learning of Driving Models from Large-scale Video Datasets
• Visual Semantic Navigation using Scene Priors
• Obstacle Avoidance through Deep Networks based Intermediate Perception
• Machine Learning Methods for Track Classification in the AT-TPC
• Rethinking Self-driving: Multi-task Knowledge for Better Generalization and Accident Explanation Ability
• Learning from Maps: Visual Common Sense for Autonomous Driving
• Aggressive Deep Driving: Model Predictive Control with a CNN Cost Model
• Curriculum Adversarial Training
• Robotic Grasp Detection using Deep Convolutional Neural Networks
• DeepSignals: Predicting Intent of Drivers Through Visual Signals
• Autonomous Braking System via Deep Reinforcement Learning
• Learning Temporal Strategic Relationships using Generative Adversarial Imitation Learning
• TorontoCity: Seeing the World with a Million Eyes
• CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion
• Model-driven Simulations for Deep Convolutional Neural Networks
• CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching
• Textual Explanations for Self-Driving Vehicles
• Gradient-free Policy Architecture Search and Adaptation
• Multi-scale Location-aware Kernel Representation for Object Detection
• Toolflows for Mapping Convolutional Neural Networks on FPGAs: A Survey and Future Directions
• Brain Inspired Cognitive Model with Attention for Self-Driving Cars
• Frame-wise Motion and Appearance for Real-time Multiple Object Tracking
• Automated vehicle's behavior decision making using deep reinforcement learning and high-fidelity simulation environment
• TKD: Temporal Knowledge Distillation for Active Perception
• LIDAR-based Driving Path Generation Using Fully Convolutional Neural Networks
• Comparing Apples and Oranges: Off-Road Pedestrian Detection on the NREC Agricultural Person-Detection Dataset
• Affordance Learning In Direct Perception for Autonomous Driving
• Modular Vehicle Control for Transferring Semantic Information Between Weather Conditions Using GANs
• Imitating Driver Behavior with Generative Adversarial Networks
• MultiNet: Multi-Modal Multi-Task Learning for Autonomous Driving
• WAD: A Deep Reinforcement Learning Agent for Urban Autonomous Driving
• Teaching UAVs to Race: End-to-End Regression of Agile Controls in Simulation
• Learning On-Road Visual Control for Self-Driving Vehicles with Auxiliary Tasks
• Deep Learning of Robotic Tasks without a Simulator using Strong and Weak Human Supervision
• A Mixture of Expert Approach for Low-Cost Customization of Deep Neural Networks
• TiEV: The Tongji Intelligent Electric Vehicle in the Intelligent Vehicle Future Challenge of China
• Quantized neural network design under weight capacity constraint
• A Self-Supervised Learning Approach to Rapid Path Planning for Car-Like Vehicles Maneuvering in Urban Environment
• End-to-End Race Driving with Deep Reinforcement Learning
• Transformer Guided Geometry Model for Flow-Based Unsupervised Visual Odometry
• Mutation Testing framework for Machine Learning
• Neural Network Activation Quantization with Bitwise Information Bottlenecks
• On Offline Evaluation of Vision-based Driving Models
• A LiDAR Assisted Control Module with High Precision in Parking Scenarios for Autonomous Driving Vehicle