Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car

Abstract

As part of a complete software stack for autonomous driving, NVIDIA has created a neural-network-based system, known as PilotNet, which outputs steering angles given images of the road ahead. PilotNet is trained using road images paired with the steering angles generated by a human driving a data-collection car. It derives the necessary domain knowledge by observing human drivers. This eliminates the need for human engineers to anticipate what is important in an image and foresee all the necessary rules for safe driving. Road tests demonstrated that PilotNet can successfully perform lane keeping in a wide variety of driving conditions, regardless of whether lane markings are present or not. The goal of the work described here is to explain what PilotNet learns and how it makes its decisions. To this end we developed a method for determining which elements in the road image most influence PilotNet's steering decision. Results show that PilotNet indeed learns to recognize relevant objects on the road. In addition to learning the obvious features such as lane markings, edges of roads, and other cars, PilotNet learns more subtle features that would be hard to anticipate and program by engineers, for example, bushes lining the edge of the road and atypical vehicle classes.

References in corpus (1)

• End to End Learning for Self-Driving Cars

Cited by in corpus (65)

• Methods for Interpreting and Understanding Deep Neural Networks
• A Survey of Deep Learning Techniques for Autonomous Driving
• Multimodal End-to-End Autonomous Driving
• A Survey of End-to-End Driving: Architectures and Training Methods
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• EmergencyNet: Efficient Aerial Image Classification for Drone-Based Emergency Monitoring Using Atrous Convolutional Feature Fusion
• Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data
• Multi-modal Sensor Fusion-Based Deep Neural Network for End-to-end Autonomous Driving with Scene Understanding
• A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving
• A Review of Sensing and Communication, Human Factors, and Controller Aspects for Information-Aware Connected and Automated Vehicles
• Machine Learning for Security in Vehicular Networks: A Comprehensive Survey
• Towards Explaining Anomalies: A Deep Taylor Decomposition of One-Class Models
• TSViz: Demystification of Deep Learning Models for Time-Series Analysis
• Controlling an Autonomous Vehicle with Deep Reinforcement Learning
• Self-Driving Car Steering Angle Prediction Based on Image Recognition
• Interpretable to Whom? A Role-based Model for Analyzing Interpretable Machine Learning Systems
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• End-to-End Deep Learning for Steering Autonomous Vehicles Considering Temporal Dependencies
• Deep Reinforcement-Learning-based Driving Policy for Autonomous Road Vehicles
• Driving Behavior Explanation with Multi-level Fusion
• End-to-end Multi-Modal Multi-Task Vehicle Control for Self-Driving Cars with Visual Perception
• Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety
• Model-free Deep Reinforcement Learning for Urban Autonomous Driving
• MADRaS : Multi Agent Driving Simulator
• Geo-Context Aware Study of Vision-Based Autonomous Driving Models and Spatial Video Data
• Interpretable End-to-end Urban Autonomous Driving with Latent Deep Reinforcement Learning
• Safe end-to-end imitation learning for model predictive control
• A Survey on Autonomous Vehicle Control in the Era of Mixed-Autonomy: From Physics-Based to AI-Guided Driving Policy Learning
• Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation
• Simple Physical Adversarial Examples against End-to-End Autonomous Driving Models
• Integrated Decision and Control at Multi-Lane Intersections with Mixed Traffic Flow
• Elements of Effective Deep Reinforcement Learning towards Tactical Driving Decision Making
• The NVIDIA PilotNet Experiments
• CortexNet: a Generic Network Family for Robust Visual Temporal Representations
• Fixed-Dimensional and Permutation Invariant State Representation of Autonomous Driving
• Enhanced Behavioral Cloning Based self-driving Car Using Transfer Learning
• Two-stream convolutional networks for end-to-end learning of self-driving cars
• Attacking Vision-based Perception in End-to-End Autonomous Driving Models
• Robustifying the Deployment of tinyML Models for Autonomous mini-vehicles
• Affordance-based Reinforcement Learning for Urban Driving
• On the Road with 16 Neurons: Mental Imagery with Bio-inspired Deep Neural Networks
• DeepRacing: Parameterized Trajectories for Autonomous Racing
• Rethinking Recurrent Neural Networks and Other Improvements for Image Classification
• Deep Object-Centric Policies for Autonomous Driving
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
• CoSA: Scheduling by Constrained Optimization for Spatial Accelerators
• Unsupervised deep learning for grading of age-related macular degeneration using retinal fundus images
• AI based Algorithms of Path Planning, Navigation and Control for Mobile Ground Robots and UAVs
• Agile Autonomous Driving using End-to-End Deep Imitation Learning
• Behavioral Research and Practical Models of Drivers' Attention
• Testing of Autonomous Driving Systems: Where Are We and Where Should We Go?
• A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle
• Cooperation-Aware Lane Change Maneuver in Dense Traffic based on Model Predictive Control with Recurrent Neural Network
• Training Interpretable Convolutional Neural Networks by Differentiating Class-specific Filters
• ML + FV = $♡$? A Survey on the Application of Machine Learning to Formal Verification
• Co-Imitation Learning without Expert Demonstration
• Dynamic Risk Assessment for Vehicles of Higher Automation Levels by Deep Learning
• Explaining Transition Systems through Program Induction
• Brain-Inspired Deep Imitation Learning for Autonomous Driving Systems
• Technical Report: When Does Machine Learning FAIL? Generalized Transferability for Evasion and Poisoning Attacks
• S2P2: Self-Supervised Goal-Directed Path Planning Using RGB-D Data for Robotic Wheelchairs
• Uncertainty-Aware Data Aggregation for Deep Imitation Learning
• Convex Online Video Frame Subset Selection using Multiple Criteria for Data Efficient Autonomous Driving
• PredCoin: Defense against Query-based Hard-label Attack
• Incorporating Orientations into End-to-end Driving Model for Steering Control