Hindsight policy gradients
arXiv:1711.06006
Abstract
A reinforcement learning agent that needs to pursue different goals across episodes requires a goal-conditional policy. In addition to their potential to generalize desirable behavior to unseen goals, such policies may also enable higher-level planning based on subgoals. In sparse-reward environments, the capacity to exploit information about the degree to which an arbitrary goal has been achieved while another goal was intended appears crucial to enable sample efficient learning. However, reinforcement learning agents have only recently been endowed with such capacity for hindsight. In this paper, we demonstrate how hindsight can be introduced to policy gradient methods, generalizing this idea to a broad class of successful algorithms. Our experiments on a diverse selection of sparse-reward environments show that hindsight leads to a remarkable increase in sample efficiency.
Accepted to ICLR 2019
References in corpus (1)
Cited by in corpus (17)
- Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
- Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
- Learning to Reach Goals via Iterated Supervised Learning
- Learning Latent Plans from Play
- Training Agents using Upside-Down Reinforcement Learning
- Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environment
- Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
- Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning
- Options as responses: Grounding behavioural hierarchies in multi-agent RL
- Counterfactual Credit Assignment in Model-Free Reinforcement Learning
- Hindsight Trust Region Policy Optimization
- Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
- Learning One Representation to Optimize All Rewards
- Exploration via Hindsight Goal Generation
- Unbiased Methods for Multi-Goal Reinforcement Learning
- Hindsight Experience Replay with Kronecker Product Approximate Curvature
- Improvements on Hindsight Learning