Hierarchical Reinforcement Learning for Self-Driving Decision-Making without Reliance on Labeled Driving Data

Abstract

Decision making for self-driving cars is usually tackled by manually encoding rules from drivers' behaviors or imitating drivers' manipulation using supervised learning techniques. Both of them rely on mass driving data to cover all possible driving scenarios. This paper presents a hierarchical reinforcement learning method for decision making of self-driving cars, which does not depend on a large amount of labeled driving data. This method comprehensively considers both high-level maneuver selection and low-level motion control in both lateral and longitudinal directions. We firstly decompose the driving tasks into three maneuvers, including driving in lane, right lane change and left lane change, and learn the sub-policy for each maneuver. Then, a master policy is learned to choose the maneuver policy to be executed in the current state. All policies including master policy and maneuver policies are represented by fully-connected neural networks and trained by using asynchronous parallel reinforcement learners (APRL), which builds a mapping from the sensory outputs to driving decisions. Different state spaces and reward functions are designed for each maneuver. We apply this method to a highway driving scenario, which demonstrates that it can realize smooth and safe decision making for self-driving cars.

Cited by in corpus (27)

• Motion Planning for Autonomous Driving: The State of the Art and Future Perspectives
• Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors
• Adaptive dynamic programming for nonaffine nonlinear optimal control problem with state constraints
• Hierarchical Program-Triggered Reinforcement Learning Agents For Automated Driving
• Relaxed Actor-Critic with Convergence Guarantees for Continuous-Time Optimal Control of Nonlinear Systems
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review
• Alternating Direction Method of Multipliers for Constrained Iterative LQR in Autonomous Driving
• Integrated Decision and Control at Multi-Lane Intersections with Mixed Traffic Flow
• Adversarial Evaluation of Autonomous Vehicles in Lane-Change Scenarios
• Fixed-Dimensional and Permutation Invariant State Representation of Autonomous Driving
• Efficient Deep Reinforcement Learning with Imitative Expert Priors for Autonomous Driving
• Optimization Landscape of Gradient Descent for Discrete-time Static Output Feedback
• Improving Generalization of Reinforcement Learning with Minimax Distributional Soft Actor-Critic
• Decision-making for Autonomous Vehicles on Highway: Deep Reinforcement Learning with Continuous Action Horizon
• Safe Reinforcement Learning for Autonomous Vehicles through Parallel Constrained Policy Optimization
• Encoding Distributional Soft Actor-Critic for Autonomous Driving in Multi-lane Scenarios
• Decision-making Strategy on Highway for Autonomous Vehicles using Deep Reinforcement Learning
• A Comparative Analysis of Deep Reinforcement Learning-enabled Freeway Decision-making for Automated Vehicles
• Decision-Making under On-Ramp merge Scenarios by Distributional Soft Actor-Critic Algorithm
• Dueling Deep Q Network for Highway Decision Making in Autonomous Vehicles: A Case Study
• Experimental Analysis of Trajectory Control Using Computer Vision and Artificial Intelligence for Autonomous Vehicles
• Feudal Steering: Hierarchical Learning for Steering Angle Prediction
• Continuous-time finite-horizon ADP for automated vehicle controller design with high efficiency
• Semi-Definite Relaxation Based ADMM for Cooperative Planning and Control of Connected Autonomous Vehicles
• Reinforcement Solver for H-infinity Filter with Bounded Noise
• Approximate Optimal Filter for Linear Gaussian Time-invariant Systems
• Model-Based Actor-Critic with Chance Constraint for Stochastic System