Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment

Abstract

This paper proposes a method for prioritizing the replay experience referred to as Hindsight Goal Ranking (HGR) in overcoming the limitation of Hindsight Experience Replay (HER) that generates hindsight goals based on uniform sampling. HGR samples with higher probability on the states visited in an episode with larger temporal difference (TD) error, which is considered as a proxy measure of the amount which the RL agent can learn from an experience. The actual sampling for large TD error is performed in two steps: first, an episode is sampled from the relay buffer according to the average TD error of its experiences, and then, for the sampled episode, the hindsight goal leading to larger TD error is sampled with higher probability from future visited states. The proposed method combined with Deep Deterministic Policy Gradient (DDPG), an off-policy model-free actor-critic algorithm, accelerates learning significantly faster than that without any prioritization on four challenging simulated robotic manipulation tasks. The empirical results show that HGR uses samples more efficiently than previous methods across all tasks.

References in corpus (21)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Dueling Network Architectures for Deep Reinforcement Learning
• End-to-End Training of Deep Visuomotor Policies
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• Benchmarking Deep Reinforcement Learning for Continuous Control
• QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
• Visual Reinforcement Learning with Imagined Goals
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning
• Automatic Goal Generation for Reinforcement Learning Agents
• Skew-Fit: State-Covering Self-Supervised Reinforcement Learning
• Maximum Entropy-Regularized Multi-Goal Reinforcement Learning
• CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning
• Hindsight policy gradients
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Generalized Hindsight for Reinforcement Learning

Cited by in corpus (1)

• Sample Efficient Reinforcement Learning via Large Vision Language Model Distillation