Composing Meta-Policies for Autonomous Driving Using Hierarchical Deep Reinforcement Learning

Abstract

Rather than learning new control policies for each new task, it is possible, when tasks share some structure, to compose a "meta-policy" from previously learned policies. This paper reports results from experiments using Deep Reinforcement Learning on a continuous-state, discrete-action autonomous driving simulator. We explore how Deep Neural Networks can represent meta-policies that switch among a set of previously learned policies, specifically in settings where the dynamics of a new scenario are composed of a mixture of previously learned dynamics and where the state observation is possibly corrupted by sensing noise. We also report the results of experiments varying dynamics mixes, distractor policies, magnitudes/distributions of sensing noise, and obstacles. In a fully observed experiment, the meta-policy learning algorithm achieves 2.6x the reward achieved by the next best policy composition technique with 80% less exploration. In a partially observed experiment, the meta-policy learning algorithm converges after 50 iterations while a direct application of RL fails to converge even after 200 iterations.

8 pages, 11 figures

References in corpus (4)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Playing Atari with Deep Reinforcement Learning
• lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits
• HIRL: Hierarchical Inverse Reinforcement Learning for Long-Horizon Tasks with Delayed Rewards

Cited by in corpus (6)

• Multi-Level Discovery of Deep Options
• Learning End-to-end Multimodal Sensor Policies for Autonomous Navigation
• Design Space of Behaviour Planning for Autonomous Driving
• Flow Rate Control in Smart District Heating Systems Using Deep Reinforcement Learning
• Learning On-Road Visual Control for Self-Driving Vehicles with Auxiliary Tasks
• Improved Robustness and Safety for Pre-Adaptation of Meta Reinforcement Learning with Prior Regularization