Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning

Abstract

What goals should a multi-goal reinforcement learning agent pursue during training in long-horizon tasks? When the desired (test time) goal distribution is too distant to offer a useful learning signal, we argue that the agent should not pursue unobtainable goals. Instead, it should set its own intrinsic goals that maximize the entropy of the historical achieved goal distribution. We propose to optimize this objective by having the agent pursue past achieved goals in sparsely explored areas of the goal space, which focuses exploration on the frontier of the achievable goal set. We show that our strategy achieves an order of magnitude better sample efficiency than the prior state of the art on long-horizon multi-goal tasks including maze navigation and block stacking.

arXiv comment12 pages (+12 appendix). Published as a conference paper at ICML 2020. Code available at https://github.com/spitis/mrl

Cited by in corpus (16)

• An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey
• MHER: Model-based Hindsight Experience Replay
• Counterfactual Data Augmentation using Locally Factored Dynamics
• Behavior From the Void: Unsupervised Active Pre-Training
• C-Learning: Learning to Achieve Goals via Recursive Classification
• Hindsight Trust Region Policy Optimization
• Hindsight Expectation Maximization for Goal-conditioned Reinforcement Learning
• A Definition of Open-Ended Learning Problems for Goal-Conditioned Agents
• C-Planning: An Automatic Curriculum for Learning Goal-Reaching Tasks
• Density-based Curriculum for Multi-goal Reinforcement Learning with Sparse Rewards
• Action Redundancy in Reinforcement Learning
• Selection for short-term empowerment accelerates the evolution of homeostatic neural cellular automata
• Stein Variational Goal Generation for adaptive Exploration in Multi-Goal Reinforcement Learning
• General Characterization of Agents by States they Visit
• Subgoal Search For Complex Reasoning Tasks
• Adaptive Multi-Goal Exploration