Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

Abstract

Autonomous driving is a multi-agent setting where the host vehicle must apply sophisticated negotiation skills with other road users when overtaking, giving way, merging, taking left and right turns and while pushing ahead in unstructured urban roadways. Since there are many possible scenarios, manually tackling all possible cases will likely yield a too simplistic policy. Moreover, one must balance between unexpected behavior of other drivers/pedestrians and at the same time not to be too defensive so that normal traffic flow is maintained. In this paper we apply deep reinforcement learning to the problem of forming long term driving strategies. We note that there are two major challenges that make autonomous driving different from other robotic tasks. First, is the necessity for ensuring functional safety - something that machine learning has difficulty with given that performance is optimized at the level of an expectation over many instances. Second, the Markov Decision Process model often used in robotics is problematic in our case because of unpredictable behavior of other agents in this multi-agent scenario. We make three contributions in our work. First, we show how policy gradient iterations can be used without Markovian assumptions. Second, we decompose the problem into a composition of a Policy for Desires (which is to be learned) and trajectory planning with hard constraints (which is not learned). The goal of Desires is to enable comfort of driving, while hard constraints guarantees the safety of driving. Third, we introduce a hierarchical temporal abstraction we call an "Option Graph" with a gating mechanism that significantly reduces the effective horizon and thereby reducing the variance of the gradient estimation even further.

Cited by in corpus (53)

• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Self-Driving Car Steering Angle Prediction Based on Image Recognition
• Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates
• MUVINE: Multi-stage Virtual Network Embedding in Cloud Data Centers using Reinforcement Learning based Predictions
• Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
• Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning
• Safe Multi-Agent Reinforcement Learning via Shielding
• Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments
• Learning Accurate, Comfortable and Human-like Driving
• Locally Private Distributed Reinforcement Learning
• Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• A Deep Ensemble Multi-Agent Reinforcement Learning Approach for Air Traffic Control
• A Finite-Time Analysis of Q-Learning with Neural Network Function Approximation
• Near-Optimal Reinforcement Learning with Self-Play
• Transferring Autonomous Driving Knowledge on Simulated and Real Intersections
• Microscopic Traffic Simulation by Cooperative Multi-agent Deep Reinforcement Learning
• Risk Averse Robust Adversarial Reinforcement Learning
• Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning
• V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• Accelerating Training in Pommerman with Imitation and Reinforcement Learning
• Learning hierarchical behavior and motion planning for autonomous driving
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
• A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
• The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
• Do Autonomous Agents Benefit from Hearing?
• Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators
• Towards General Function Approximation in Zero-Sum Markov Games
• Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applications
• Cooperation-Aware Lane Change Maneuver in Dense Traffic based on Model Predictive Control with Recurrent Neural Network
• VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control
• Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning
• Decentralized Multi-Agent Reinforcement Learning for Task Offloading Under Uncertainty
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
• Approaching Neural Network Uncertainty Realism
• Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
• Expressing Diverse Human Driving Behavior with Probabilistic Rewards and Online Inference
• Multi-Agent Reinforcement Learning in Cournot Games
• Reinforcement Learning for Multi-Objective Optimization of Online Decisions in High-Dimensional Systems
• Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT
• An NCAP-like Safety Indicator for Self-Driving Cars
• Spatio-Temporal Graph Scattering Transform
• Parameter Critic: a Model Free Variance Reduction Method Through Imperishable Samples
• Towards Brain-inspired System: Deep Recurrent Reinforcement Learning for Simulated Self-driving Agent
• Two-stage training algorithm for AI robot soccer
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• Learning Cooperation and Online Planning Through Simulation and Graph Convolutional Network
• Safe Reinforcement Learning on Autonomous Vehicles
• The Power of Communication in a Distributed Multi-Agent System