Hindsight Trust Region Policy Optimization
arXiv:1907.12439
Abstract
Reinforcement Learning(RL) with sparse rewards is a major challenge. We propose \emph{Hindsight Trust Region Policy Optimization}(HTRPO), a new RL algorithm that extends the highly successful TRPO algorithm with \emph{hindsight} to tackle the challenge of sparse rewards. Hindsight refers to the algorithm's ability to learn from information across goals, including ones not intended for the current task. HTRPO leverages two main ideas. It introduces QKL, a quadratic approximation to the KL divergence constraint on the trust region, leading to reduced variance in KL divergence estimation and improved stability in policy update. It also presents Hindsight Goal Filtering(HGF) to select conductive hindsight goals. In experiments, we evaluate HTRPO in various sparse reward tasks, including simple benchmarks, image-based Atari games, and simulated robot control. Ablation studies indicate that QKL and HGF contribute greatly to learning stability and high performance. Comparison results show that in all tasks, HTRPO consistently outperforms both TRPO and HPG, a state-of-the-art algorithm for RL with sparse rewards.
Accepted by IJCAI 2021
References in corpus (9)
- Emergence of Locomotion Behaviours in Rich Environments
- Visual Reinforcement Learning with Imagined Goals
- Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic
- Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
- Hierarchical Reinforcement Learning with Hindsight
- Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
- Hindsight policy gradients
- Many-Goals Reinforcement Learning
- Hierarchical Imitation and Reinforcement Learning