Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic

Abstract

Learning a policy using only observational data is challenging because the distribution of states it induces at execution time may differ from the distribution observed during training. We propose to train a policy by unrolling a learned model of the environment dynamics over multiple time steps while explicitly penalizing two costs: the original cost the policy seeks to optimize, and an uncertainty cost which represents its divergence from the states it is trained on. We measure this second cost by using the uncertainty of the dynamics model about its own predictions, using recent ideas from uncertainty estimation for deep networks. We evaluate our approach using a large-scale observational dataset of driving behavior recorded from traffic cameras, and show that we are able to learn effective driving policies from purely observational data, with no environment interaction.

References in corpus (13)

• Auto-Encoding Variational Bayes
• Improving neural networks by preventing co-adaptation of feature detectors
• End to End Learning for Self-Driving Cars
• Stochastic Video Generation with a Learned Prior
• Uncertainty-Aware Reinforcement Learning for Collision Avoidance
• Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control
• Imagination-Augmented Agents for Deep Reinforcement Learning
• Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
• Decomposition of Uncertainty in Bayesian Deep Learning for Efficient and Risk-sensitive Learning
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Learning model-based planning from scratch
• Stochastic Variational Video Prediction
• Learning Simple Algorithms from Examples

Cited by in corpus (30)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• A Survey of Deep RL and IL for Autonomous Driving Policy Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Estimating Risk and Uncertainty in Deep Reinforcement Learning
• Simple and Effective VAE Training with Calibrated Decoders
• Bridging adaptive management and reinforcement learning for more robust decisions
• Rethinking Closed-loop Training for Autonomous Driving
• Deliberative Acting, Online Planning and Learning with Hierarchical Operational Models
• Dynamics-aware Embeddings
• Deep Constrained Q-learning
• Model-based Reinforcement Learning for Predictions and Control for Limit Order Books
• Learning the smoothness of noisy curves with application to online curve estimation
• CARLA Real Traffic Scenarios -- novel training ground and benchmark for autonomous driving
• Batch Policy Learning under Constraints
• Reset-Free Lifelong Learning with Skill-Space Planning
• Triple-GAIL: A Multi-Modal Imitation Learning Framework with Generative Adversarial Nets
• SAFARI: Safe and Active Robot Imitation Learning with Imagination
• Improving Robustness of Deep Neural Networks for Aerial Navigation by Incorporating Input Uncertainty
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors
• Learning to Predict Navigational Patterns from Partial Observations
• Visual-based Autonomous Driving Deployment from a Stochastic and Uncertainty-aware Perspective
• MIDAS: Multi-agent Interaction-aware Decision-making with Adaptive Strategies for Urban Autonomous Navigation
• SeRO: Self-Supervised Reinforcement Learning for Recovery from Out-of-Distribution Situations
• Mitigating the Effects of Non-Identifiability on Inference for Bayesian Neural Networks with Latent Variables
• Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles
• Multi-Turn Beam Search for Neural Dialogue Modeling
• UMBRELLA: Uncertainty-Aware Model-Based Offline Reinforcement Learning Leveraging Planning
• What classifiers know what they don't?
• Understanding the Role of Adversarial Regularization in Supervised Learning
• Evaluating task-agnostic exploration for fixed-batch learning of arbitrary future tasks