Safe, Multi-Agent, Reinforcement Learning for Autonomous Driving

Abstract

Autonomous driving is a multi-agent setting where the host vehicle must apply sophisticated negotiation skills with other road users when overtaking, giving way, merging, taking left and right turns and while pushing ahead in unstructured urban roadways. Since there are many possible scenarios, manually tackling all possible cases will likely yield a too simplistic policy. Moreover, one must balance between unexpected behavior of other drivers/pedestrians and at the same time not to be too defensive so that normal traffic flow is maintained. In this paper we apply deep reinforcement learning to the problem of forming long term driving strategies. We note that there are two major challenges that make autonomous driving different from other robotic tasks. First, is the necessity for ensuring functional safety - something that machine learning has difficulty with given that performance is optimized at the level of an expectation over many instances. Second, the Markov Decision Process model often used in robotics is problematic in our case because of unpredictable behavior of other agents in this multi-agent scenario. We make three contributions in our work. First, we show how policy gradient iterations can be used without Markovian assumptions. Second, we decompose the problem into a composition of a Policy for Desires (which is to be learned) and trajectory planning with hard constraints (which is not learned). The goal of Desires is to enable comfort of driving, while hard constraints guarantees the safety of driving. Third, we introduce a hierarchical temporal abstraction we call an "Option Graph" with a gating mechanism that significantly reduces the effective horizon and thereby reducing the variance of the gradient estimation even further.

References in corpus (4)

• End to End Learning for Self-Driving Cars
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• SDCA without Duality, Regularization, and Individual Convexity

Cited by in corpus (140)

• A Survey of Deep Learning Techniques for Autonomous Driving
• On a Formal Model of Safe and Scalable Self-driving Cars
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving
• A Survey of End-to-End Driving: Architectures and Training Methods
• Multi-agent Reinforcement Learning for Cooperative Lane Changing of Connected and Autonomous Vehicles in Mixed Traffic
• Virtual to Real Reinforcement Learning for Autonomous Driving
• Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge
• Game-Theoretic Multiagent Reinforcement Learning
• Variational End-to-End Navigation and Localization
• SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Driving
• Driving in Dense Traffic with Model-Free Reinforcement Learning
• Self-Driving Car Steering Angle Prediction Based on Image Recognition
• Barrier-Certified Adaptive Reinforcement Learning with Applications to Brushbot Navigation
• Cooperative and Distributed Reinforcement Learning of Drones for Field Coverage
• Ensemble Quantile Networks: Uncertainty-Aware Reinforcement Learning with Applications in Autonomous Driving
• Accelerated Primal-Dual Policy Optimization for Safe Reinforcement Learning
• Adaptive Behavior Generation for Autonomous Driving using Deep Reinforcement Learning with Compact Semantic States
• Driving with Style: Inverse Reinforcement Learning in General-Purpose Planning for Automated Driving
• Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear
• Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
• Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates
• Deep Reinforcement-Learning-based Driving Policy for Autonomous Road Vehicles
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• MUVINE: Multi-stage Virtual Network Embedding in Cloud Data Centers using Reinforcement Learning based Predictions
• Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Bound
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations
• Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
• DeepTraffic: Crowdsourced Hyperparameter Tuning of Deep Reinforcement Learning Systems for Multi-Agent Dense Traffic Navigation
• Addressing Inherent Uncertainty: Risk-Sensitive Behavior Generation for Automated Driving using Distributional Reinforcement Learning
• Graph Convolutional Reinforcement Learning
• Safe Multi-Agent Reinforcement Learning via Shielding
• Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension
• Finite-Sample Analysis of Nonlinear Stochastic Approximation with Applications in Reinforcement Learning
• MADRaS : Multi Agent Driving Simulator
• Succinct and Robust Multi-Agent Communication With Temporal Message Control
• Autonomous Driving in Reality with Reinforcement Learning and Image Translation
• Evolutionary Reinforcement Learning for Sample-Efficient Multiagent Coordination
• Is Pessimism Provably Efficient for Offline RL?
• Model-Based Multi-Agent RL in Zero-Sum Markov Games with Near-Optimal Sample Complexity
• Combining Neural Networks and Tree Search for Task and Motion Planning in Challenging Environments
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Self-driving scale car trained by Deep reinforcement learning
• Learning Accurate, Comfortable and Human-like Driving
• Autonomous Ramp Merge Maneuver Based on Reinforcement Learning with Continuous Action Space
• Locally Private Distributed Reinforcement Learning
• Actor-Critic Provably Finds Nash Equilibria of Linear-Quadratic Mean-Field Games
• Learning Individually Inferred Communication for Multi-Agent Cooperation
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Lane-Merging Using Policy-based Reinforcement Learning and Post-Optimization
• QTRAN++: Improved Value Transformation for Cooperative Multi-Agent Reinforcement Learning
• Building Safer Autonomous Agents by Leveraging Risky Driving Behavior Knowledge
• A Deep Ensemble Multi-Agent Reinforcement Learning Approach for Air Traffic Control
• Multi-Agent Constrained Policy Optimisation
• A Finite-Time Analysis of Q-Learning with Neural Network Function Approximation
• Stochastic Recursive Momentum for Policy Gradient Methods
• Transferring Autonomous Driving Knowledge on Simulated and Real Intersections
• Near-Optimal Reinforcement Learning with Self-Play
• Elements of Effective Deep Reinforcement Learning towards Tactical Driving Decision Making
• Nonstationary Reinforcement Learning with Linear Function Approximation
• Finite-Sample Analysis of Stochastic Approximation Using Smooth Convex Envelopes
• Planning on the fast lane: Learning to interact using attention mechanisms in path integral inverse reinforcement learning
• SAM: Squeeze-and-Mimic Networks for Conditional Visual Driving Policy Learning
• Risk Averse Robust Adversarial Reinforcement Learning
• V-Learning -- A Simple, Efficient, Decentralized Algorithm for Multiagent RL
• Analyzing Knowledge Transfer in Deep Q-Networks for Autonomously Handling Multiple Intersections
• Extended Markov Games to Learn Multiple Tasks in Multi-Agent Reinforcement Learning
• Microscopic Traffic Simulation by Cooperative Multi-agent Deep Reinforcement Learning
• Decentralized Control of Quadrotor Swarms with End-to-end Deep Reinforcement Learning
• Design Space of Behaviour Planning for Autonomous Driving
• Implicit Multiagent Coordination at Unsignalized Intersections via Multimodal Inference Enabled by Topological Braids
• Decentralized Q-Learning in Zero-sum Markov Games
• Natural Actor-Critic Converges Globally for Hierarchical Linear Quadratic Regulator
• Fever Basketball: A Complex, Flexible, and Asynchronized Sports Game Environment for Multi-agent Reinforcement Learning
• BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning
• Learning hierarchical behavior and motion planning for autonomous driving
• Accelerating Training in Pommerman with Imitation and Reinforcement Learning
• On Reinforcement Learning for Full-length Game of StarCraft
• PowerNet: Multi-agent Deep Reinforcement Learning for Scalable Powergrid Control
• Global Convergence of Policy Gradient for Linear-Quadratic Mean-Field Control/Game in Continuous Time
• A Survey of Deep Reinforcement Learning Algorithms for Motion Planning and Control of Autonomous Vehicles
• Reinforcement Learning Based Safe Decision Making for Highway Autonomous Driving
• Decision Making for Autonomous Driving via Augmented Adversarial Inverse Reinforcement Learning
• A Visual Communication Map for Multi-Agent Deep Reinforcement Learning
• Do Autonomous Agents Benefit from Hearing?
• Modelling Bounded Rationality in Multi-Agent Interactions by Generalized Recursive Reasoning
• The Power of Exploiter: Provable Multi-Agent RL in Large State Spaces
• Learning Zero-Sum Simultaneous-Move Markov Games Using Function Approximation and Correlated Equilibrium
• Reinforcement Learning based Control of Imitative Policies for Near-Accident Driving
• Reinforcement Learning for Joint Optimization of Multiple Rewards
• Towards General Function Approximation in Zero-Sum Markov Games
• Finite-Sample Analysis of Off-Policy TD-Learning via Generalized Bellman Operators
• Independent Learning in Stochastic Games
• CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing
• Joint Attention for Multi-Agent Coordination and Social Learning
• Towards a Systematic Computational Framework for Modeling Multi-Agent Decision-Making at Micro Level for Smart Vehicles in a Smart World
• Permutation Invariant Policy Optimization for Mean-Field Multi-Agent Reinforcement Learning: A Principled Approach
• Decentralized Multi-Agent Reinforcement Learning for Task Offloading Under Uncertainty
• Dimension-Free Rates for Natural Policy Gradient in Multi-Agent Reinforcement Learning
• CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving
• VMAV-C: A Deep Attention-based Reinforcement Learning Algorithm for Model-based Control
• Counterfactual Policy Evaluation for Decision-Making in Autonomous Driving
• Mean-Field Multi-Agent Reinforcement Learning: A Decentralized Network Approach
• Cooperation-Aware Lane Change Maneuver in Dense Traffic based on Model Predictive Control with Recurrent Neural Network
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Communication-Efficient Zeroth-Order Distributed Online Optimization: Algorithm, Theory, and Applications
• Cutting Your Losses: Learning Fault-Tolerant Control and Optimal Stopping under Adverse Risk
• Off-Policy Exploitability-Evaluation in Two-Player Zero-Sum Markov Games
• Approaching Neural Network Uncertainty Realism
• Multi-Agent Reinforcement Learning in Cournot Games
• A Survey of FPGA-Based Robotic Computing
• Virtuously Safe Reinforcement Learning
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Fully Bayesian Recurrent Neural Networks for Safe Reinforcement Learning
• Human and Multi-Agent collaboration in a human-MARL teaming framework
• Reinforcement Learning for Multi-Objective Optimization of Online Decisions in High-Dimensional Systems
• Expressing Diverse Human Driving Behavior with Probabilistic Rewards and Online Inference
• Policy Gradient Methods Find the Nash Equilibrium in N-player General-sum Linear-quadratic Games
• Bregman Gradient Policy Optimization
• PowerGridworld: A Framework for Multi-Agent Reinforcement Learning in Power Systems
• Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge Industrial IoT
• Safe learning-based optimal motion planning for automated driving
• Constrained Policy Improvement for Safe and Efficient Reinforcement Learning
• An NCAP-like Safety Indicator for Self-Driving Cars
• Single-step Options for Adversary Driving
• Online Learning in Unknown Markov Games
• State-based Episodic Memory for Multi-Agent Reinforcement Learning
• Spatio-Temporal Graph Scattering Transform
• The Power of Communication in a Distributed Multi-Agent System
• Towards Brain-inspired System: Deep Recurrent Reinforcement Learning for Simulated Self-driving Agent
• Two-stage training algorithm for AI robot soccer
• VisualEnv: visual Gym environments with Blender
• A Law of Iterated Logarithm for Multi-Agent Reinforcement Learning
• Parameter Critic: a Model Free Variance Reduction Method Through Imperishable Samples
• Safe Reinforcement Learning on Autonomous Vehicles
• A Policy Efficient Reduction Approach to Convex Constrained Deep Reinforcement Learning
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation
• Collaborative Planning for Mixed-Autonomy Lane Merging
• Learning Cooperation and Online Planning Through Simulation and Graph Convolutional Network
• Simulated Data Generation Through Algorithmic Force Coefficient Estimation for AI-Based Robotic Projectile Launch Modeling