Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
arXiv:2006.07549
Abstract
We propose a graphical model framework for goal-conditioned RL, with an EM algorithm that operates on the lower bound of the RL objective. The E-step provides a natural interpretation of how 'learning in hindsight' techniques, such as HER, to handle extremely sparse goal-conditioned rewards. The M-step reduces policy optimization to supervised learning updates, which greatly stabilizes end-to-end training on high-dimensional inputs such as images. We show that the combined algorithm, hEM significantly outperforms model-free baselines on a wide range of goal-conditioned benchmarks with sparse rewards.
arXiv commentAccepted at International Conference on Artificial Intelligence and Statistics (AISTATS), 2021
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- Continuous control with deep reinforcement learning
- Playing Atari with Deep Reinforcement Learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Prioritized Experience Replay
- Noisy Networks for Exploration
- Parameter Space Noise for Exploration
- Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
- Visual Reinforcement Learning with Imagined Goals
- Maximum a Posteriori Policy Optimisation
- Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels
- Latent Space Policies for Hierarchical Reinforcement Learning
- Taming the Noise in Reinforcement Learning via Soft Updates
- Planning with Goal-Conditioned Policies
- Generative Adversarial Self-Imitation Learning
- V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
- Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning
- Hindsight policy gradients
- Monte-Carlo Tree Search as Regularized Policy Optimization
- Automatic Curriculum Learning through Value Disagreement
- Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
- Tighter Variational Bounds are Not Necessarily Better
- Learning to Reach Goals via Iterated Supervised Learning
- Recall Traces: Backtracking Models for Efficient Reinforcement Learning
- Self-Imitation Learning via Generalized Lower Bound Q-learning